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Abstract 

This paper is focused on the performance analysis of binary linear block codes (or ensembles) whose 
transmission takes place over independent and mcmorylcss parallel channels. New upper bounds on the 
maximum-likelihood (ML) decoding error probability are derived. These bounds are applied to various en- 
sembles of turbo-like codes, focusing especially on repeat-accumulate codes and their recent variations which 
possess low encoding and decoding complexity and exhibit remarkable performance under iterative decoding. 
The framework of the second version of the Duman and Salehi (DS2) bounds is generalized to the case of 
parallel channels, along with the derivation of their optimized tilting measures. The connection between the 
generalized DS2 and the f 961 Gallager bounds, addressed by Divsalar and by Sason and Shamai for a single 
channel, is explored in the case of an arbitrary number of independent parallel channels. The generalization 
of the DS2 bound for parallel channels enables to re-derive specific bounds which were originally derived 
by Liu et al. as special cases of the Gallager bound. In the asymptotic case where we let the block length 
tend to infinity, the new bounds are used to obtain improved inner bounds on the attainable channel regions 
under ML decoding. The tightness of the new bounds for independent parallel channels is exemplified for 
structured ensembles of turbo-like codes. The improved bounds with their optimized tilting measures show, 
irrespectively of the block length of the codes, an improvement over the union bound and other previously 
reported bounds for independent parallel channels; this improvement is especially pronounced for moderate 
to large block lengths. However, in some cases, the new bounds under ML decoding happen to be a bit 
pessimistic as compared to computer simulations of sub-optimal iterative decoding, thus indicating that 
there is room for further improvement. 



Index Terms: Block codes, distance spectrum, input-output weight enumerator (IOWE), linear 
codes, maximum-likelihood (ML) decoding, memoryless binary-input output-symmetric (MBIOS) 
channels, parallel channels, repeat-accumulate (RA) codes. 
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1 Introduction 



We analyze the error performance of linear codes where the codewords are partitioned into several 
disjoint subsets, and the bits in each subset are transmitted over a certain communication channel. 
This scenario can be viewed as the transmission of information over a set of parallel channels, where 
every bit is assigned to one of these channels. Code partitioning is employed in transmission over 
block-fading channels (for performance bounds of coded communication systems over block-fading 
channels, see, e.g., [HJ ES]); rate-compatible puncturing of turbo-like codes (see, e.g., l3T]). 
incremental redundancy retransmission schemes, cooperative coding, multi-carrier signaling (for 
performance bounds of coded orthogonal-frequency division multiplexing (OFDM) systems, see 
e.g., J4J), and other applications. 

In his thesis Ebert considered the problem of communicating over parallel discrete-time 
channels, disturbed by arbitrary and independent additive Gaussian noises, where a total power 
constraint is imposed upon the channel inputs. He found explicit upper and lower bounds on the 
ML decoding error probability, which decrease exponentially with block length. The exponents of 
the upper and lower bounds coincide for zero rate and for rates between the critical rate (-R C rit) and 
capacity. The results were also shown to be applicable to colored Gaussian noise channels with an 
average power constraint on the channel. 

Tight analytical bounds serve as a potent tool for assessing the performance of modern error- 
correction schemes, both for the case of finite block length and in the asymptotic case where the 
block length tends to infinity. In the setting of a single communication channel and by letting the 
block length tend to infinity, these bounds are applied in order to obtain a noise threshold which 
indicates the minimum channel conditions necessary for reliable communication. When generalizing 
the bounds to the scenario of independent parallel channels, this threshold is transformed into a 
multi-dimensional barrier within the space of the joint parallel-channel transition probabilities, 
dividing the space into attainable and non- attainable channel regions. 

In [2^, Liu et al. derive upper bounds on the ML decoding error probability of structured 
ensembles of codes whose transmission takes place over (independent) parallel channels. When 
generalizing an upper bound to the case of independent parallel channels, it is desirable to have 
the resulting bound expressed in terms of basic features of the code (or ensemble of codes), such as 
the distance spectrum. The inherent asymmetry of the parallel-channel setting poses a difficulty 
for the analysis, as different symbols of the codeword suffer varying degrees of degradation through 
the different parallel channels. This difficulty was circumvented in |21[ by introducing a random 
mapper, i.e., a device which randomly and independently assigns symbols to the different channels 
according to a certain a-priori probability distribution. As a result of this randomization, Liu 
et al. derived upper bounds on the ML decoding error probability which solely depend on the 
weight enumerator of the overall code, instead of a specific split weight enumerator which follows 
from the partitioning of a codeword into several subsets of bits and the individual transmission 
of these subsets over different channels. The analysis in modifies the 1961 Gallager-Fano 
bound from 12, Chapter 3] and adapts this bounding technique for communication over parallel 
channels. As special cases of this modified bound, a generalization of the union-Bhattacharyya 
bound, the Shulman-Feder bound simplified sphere bound [S], and a combination of the two 
former bounds are derived for parallel channels. The upper bounds on the ML decoding error 
probability are applied to ensembles of codes defined on graphs (e.g., uniformly interleaved repeat- 
accumulate codes and turbo codes). The comparison in [2^ between upper bounds under ML 
decoding and computer simulations of the performance of such ensembles under iterative decoding 
shows a good match in several cases. For a given ensemble of codes and a given codeword-symbol 
to channel assignment rule, a reliable channel region is defined as the closure of the set of parallel- 
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channel transition probabilities for which the decoding error probability vanishes as the codeword 
length goes to infinity. The upper bounds on the block error probability derived in enable to 
derive achievable regions for ensuring reliable communications under ML decoding. 

Using the approach of the random mapper by Liu et al. [U, we derive a parallel-channel 
generalization of the DS2 bound O I29| I32| and re-examine, for the case of parallel channels, 
the well-known relations between this bound and the 1961 Gallager bound which exist for the 
single channel case (El E2]- In this respect, it is shown in this paper that the generalization of 
the DS2 bound for independent parallel channels is not necessarily tighter than the corresponding 
generalization of the 1961 Gallager bound, as opposed to the case where the communication takes 
place over a single memoryless binary-input output-symmetric (MBIOS) channel. 

The new bounds are used to obtain inner bounds on the boundary of the channel regions which 
are asymptotically (in the limit where we let the block length tend to infinity) attainable under ML 
decoding, and the results improve on those recently reported in |2j. The generalization of the DS2 
bound for parallel channels enables to reproduce special instances which were originally derived 
by Liu et al. [2j|. The tightness of these bounds for independent parallel channels is exemplified 
for structured ensembles of turbo-like codes, and the boundary of the improved attainable channel 
regions is compared with previously reported regions for Gaussian parallel channels, and shows 
significant improvement due the optimization of the tilting measures which are involved in the 
computation of the generalized DS2 and 1961 Gallager bounds for parallel channels. 

The remainder of the paper is organized as follows. The system model is presented in Section^ 
as well as preliminary material related to our discussion. In Section |31 we generalize the DS2 
bound for independent parallel channels. Section 0] presents the 1961 Gallager bound from [2*T] . 
and considers its connection to the generalized DS2 bound, along with the optimization of its 
tilting measures. Section \5\ presents some special cases of these upper bounds which are obtained 
as particular cases of the generalized bounds in Sections El and EI Attainable channel regions are 
derived in Section H3 Inner bounds on attainable channel regions for various ensembles of turbo-like 
codes and performance bounds for moderate block lengths are exemplified in Section EJ Finally, 
Section El concludes the paper. For a comprehensive tutorial paper on performance bounds of linear 
codes under ML decoding, the reader is referred to |29| . 

2 Preliminaries 

In this section, we state the assumptions on which our analysis is based. We also introduce no- 
tation and preliminary material related to the performance analysis of binary linear codes whose 
transmission takes place over parallel channels. 

2.1 System Model 

We consider the case where the communication model consists of a parallel concatenation of J 
statistically independent MBIOS channels, as shown in Fig. ^ 

Using an error-correcting linear code C of size M = 2 fc , the encoder selects a codeword x m 
(m = 0, 1, . . . , M — 1) to be transmitted, where all codewords are assumed to be selected with 
equal probability (jj)- Each codeword consists of n symbols and the coding rate is defined as 
R — log ^ ~ = this setting is referred to as using an (n, k) code. The channel mapper selects for 
each coded symbol one of J channels through which it is transmitted. The j-th channel component 
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Figure 1: System model of parallel channels. A random mapper is assumed where every bit is 
assigned to one of the J channels; a bit is assigned to the j th channel independently of the other 
bits and with probability ctj (where Ylj=i a j = !)• 



has a transition probability p(y\x;j). The considered model assumes that the channel encoder 
performs its operation without prior knowledge of the specific mapping of the bits to the parallel 
channels. While in reality, the choice of the specific mapping is subject to the levels of importance 
of different coded bits, the considered model assumes for the sake of analysis that this mapping is 
random and independent of the coded bits. This assumption enables to average over all possible 
mappings, though suitable choices of mappings for the coded bits are expected to perform better 
than the average. 

The received vector y is maximum-likelihood (ML) decoded at the receiver when the specific 
channel mapper is known at the receiver. While this broad setting gives rise to very general coding, 
mapping and decoding schemes, we will focus on the case where the input alphabet is binary, i.e., 
x G {—1, 1} (where zero and one are mapped to +1 and —1, respectively). The output alphabet is 
real, and may be either finite or continuous. By its definition, the mapping device divides the set 
of indices {1, . . . , n} into J disjoint subsets for j = 1, . . . , J, and transmits all the bits whose 
indices are included in the subset T{j) through the j-th channel. We will see in the next section 
that for a fixed channel mapping device (i.e., for given sets the problem of upper-bounding 

the ML decoding error probability is exceedingly difficult. In order to circumvent this difficulty, a 
probabilistic mapping device was introduced in 21: which uses a random assignment of the bits 
to the J parallel channels; this random mapper takes a symbol and assigns it to channel j with 
probability ay. This assignment is independent of that of other symbols, and by definition, the 
equality Ylj=i a j = 1 follows. This approach enables in [2^ the derivation of an upper bound for 
the parallel channels which is averaged over all possible channel assignments, and the bound can 
be calculated in terms of the distance spectrum of the code (or ensemble). Another benefit of the 
random mapping approach is that it naturally accommodates for practical settings where one is 
faced with parallel channels having different capacities. 



2.2 Capacity Limit and Cutoff Rate of Parallel MBIOS Channels 

We consider here the capacity and cutoff rate of independent parallel MBIOS channels. These 
information-theoretic quantities serve as a benchmark for assessing the gap under optimal ML 
decoding between the achievable channel regions of various ensembles of codes and the achievable 
channel region which corresponds to the Shannon capacity limit. It is also useful for providing 
a quantitative measure for the asymptotic performance of various ensembles as compared to the 
achievable channel region which corresponds to the cutoff rate of the considered parallel channels. 
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2.2.1 Cutoff Rate 



The cutoff rate of an MBIOS channel is given by 

i?o = l-log 2 (l+ 7 ) (1) 
where 7 is the Bhattacharyya constant, i.e., 

7 = £v»P). (2) 

y 

Clearly, for continuous-output channels, the sum in the RHS of (J2J) is replaced by an integral. 

For parallel MBIOS channels where every bit is assumed to be independently and randomly 
assigned to one of J channels with a-priori probability ay (where Yuj=i a j = -Qj the Bhattacharyya 
constant of the resulting channel is equal to the weighted sum of the Bhattacharyya constants of 
these individual channels, i.e., 

J 

7 = ^2 a 3 Vp(y\°d)p(y\hj)- (3) 

J=i y 

Consider a set of J parallel Gaussian channels characterized by the transition probabilities 



p(y\0; j) 



1 <y+^ 2 



2ir 

1 (V-Vjf 



= -7=e 2 (4) 



-co < y < oo, j = 1,..., J 

where 

and (^-j^J is the energy per information bit to the one-sided spectral noise density of the j-th 
channel. In this case, the Bhattacharyya constant is given by 

where Vj is introduced in ©. From Q and ©, the cutoff rate of J parallel binary-input AWGN 
channels is given by 

( J ~ R (—) \ 
R = l- log 2 1 + a J e 3 bits 

per channel use. (7) 

V i=i / 

Consider the case of J = 2 parallel binary-input AWGN channels. Given the value of ( jP- ) , and 



the code rate R (in bits per channel use), it is possible to calculate the value of [jfaj of the 

second channel which corresponds to the cutoff rate. To this end, we set i?o in the LHS of to 
R. Solving this equation gives 

W ) 2 = -R ln [ a 2 I" (8) 
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2.2.2 Capacity Limit 



Let Cj designate the capacity (in bits per channel use) of the j-th. MBIOS channel the set of 
J parallel MBIOS channels. Clearly, by symmetry considerations, the capacity-achieving input 
distribution for all these channels is q = (g, 2)- ^he ca P acr ty of the J parallel channels where each 
bit is randomly and independently assigned to the j-th. channel with probability ay is therefore 
given by 

J 

C = J2<*jCj. (9) 

i=i 

For the case of J parallel binary-input AWGN channels 




(E b /N o ) 1 [dB] 



Figure 2: Attainable channel regions for two parallel binary-input AWGN channels, as determined 
by the cutoff rate and the capacity limit, referring to a code rate of one-third bits per channel use. 
It is assumed that each bit is randomly and independently assigned to one of these channels with 
equal probability (i.e., a\ = «2 = 2")- 

In order to simplify the numerical computation of the capacity, one can express each integral in 
()1U|) as a sum of two integrals from to 00, and use the power series expansion of the logarithmic 
function; this gives an infinite power series with alternating signs. Using the Euler transform to 
expedite the convergence rate of these infinite sums, gives the following alternative expression: 

j = l,...,J (11) 



2j3e~ 



ln(2) 



2tt 



(2/3? 



fc=0 



(-l) fc • A k a (j) 
2 k+i 
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where 



AaoO) = -e 2 L( (fc _ m + 1)(fc _ m + 2) 

m=0 

and 

erfcx(x) = 2e x2 Q(v / 2~r) 

(note that erfcx(x) ~ -4= • - for large values of x). The infinite sum in (1111) converges exponentially 
fast with fc, and the summation of its first 30 terms happens to give very accurate result irrespective 
of the value of j3j . 

Consider again the case of J = 2 parallel binary- input AWGN channels. Given the value of 
("fo) ' an d the code rate R (in bits per channel use), and (|TT)|) enable one to calculate the value 




for the second channel, referring to the capacity limitation. To this end, one needs to set 



C in the LHS of © to the code rate R, and find the resulting value of \ which corresponds 
to the capacity limit. The boundary of the attainable channel region referring to the capacity limit 
is represented by the continuous curve in Fig. H for R = I bits per channel use; it is compared 
to the dashed curve in this figure which represents the boundary of the attainable channel region 
referring to the cutoff rate limit (see Eq. (JEJ))- 



in 



erfcx 



(2k - 2m + 3)/3j 
V2 



2.3 Distance Properties of Ensembles of Turbo-Like Codes 

In this paper, we exemplify our numerical results by considering several ensembles of binary linear 
codes. Due to the outstanding performance of turbo-like codes, our focus is mainly on such ensem- 
bles, where we also consider as a reference the ensemble of fully random block codes which achieves 
capacity under ML decoding. The other ensembles considered in this paper include turbo codes, 
repeat-accumulate codes and some recent variations. 

Bounds on the ML decoding error probability are often based on the distance properties of the 
considered codes or ensembles (see, e.g., (25] and references therein). The distance spectra and 
their asymptotic growth rates for various turbo-like ensembles were studied in the literature, e.g., 
for ensembles of uniformly interleaved repeat-accumulate codes and variations [Tj EH ; ensembles 
of uniformly interleaved turbo codes [HI EJ EH] 00] , and ensembles of regular and irregular LDPC 
codes (SJ H3 ED HH ■ I n this subsection, we briefly present the distance properties of some turbo-like 
ensembles considered in this paper. 

Let us denote by [C(n)] an ensemble of codes of length n. We will also consider a sequence of 
ensembles [C(ni)], [C(ri2)], ■ ■ ■ where all these ensembles possess a common rate R. For a given (n, k) 
linear code C, let (or simply Ah) denote the distance spectrum, i.e., the number of codewords 
of Hamming weight h. For a set of codes [C(n)], we define the average distance spectrum as 

A [C(n)] a _j_ ,C h?) 

A h - | [c(n)] | 2^ A h- ( u ) 

11 v > n Ce[C(n)] 

Let ^ n = {5 : 5 = ^ for h = 1, . . . ,n} = {^,^,...,1} denote the set of normalized distances, then 
the normalized exponent of the distance spectrum w.r.t. the block length is defined as 

In A c In 4l c ( n )] 

r [C(n)I (< y) A ^!_. (13) 

n n 



7 



The motivation for this definition lies in the interest to consider the asymptotic case where n — > oo. 
In this case we define the asymptotic exponent of the distance spectrum as 

A C \S) = lim A C ^(S) . (14) 

n^oo 



The input-output weight enumerator (IOWE) of a linear block code is given by a sequence 
{A Wt h} designating the number of codewords of Hamming weight h which are encoded by informa- 
tion bits whose Hamming weight is w. Referring to ensembles, one considers the average IOWE 
and distance spectrum over the ensemble. The distance spectrum and the IOWE of linear block 
codes are useful for the analysis of the block and bit error probabilities, respectively, under ML 
decoding. 

As a reference to all ensembles, we will consider the ensemble of fully random block codes which 
is capacity-achieving under ML decoding (or 'typical pairs') decoding. 

The ensemble of fully random binary linear block codes: Consider the ensemble of binary random 
codes [TZB], where the set [lZB(n, R)] consists of all binary codes of length n and rate R. For this 
ensemble, the following well-known equalities hold: 

A [KB(n,R)] fn\ -n(l-R) 

- w 

r [KB(n,R)]f 6) = ^_(l_^)l n2 (15) 
n 



,[RJ3(R)] 



(5) = H(S) - (1-J2)ln2 



where H(x) = — xln(x) — (1 — x)ln(l — x) designates the binary entropy function to the natural 
base. 

N on- systematic repeat- accumulate codes: The ensemble of uniformly interleaved and non-systematic 
repeat-accumulate (NSRA) codes is defined as follows. The information block of length N is 
repeated q times by the encoder. The bits are then uniformly reordered by an interleaver of size 
qN, and, finally, encoded by a rate-1 differential encoder (accumulator), i.e., a truncated rate-1 
recursive convolutional encoder with a transfer function 1/(1 + D). The ensemble [NSRA(A, q)\ is 
defined to be the set of RA different codes when considering the different possible permuta- 

tions of the interleaver. 1 The (average) IOWE of the ensemble of uniformly interleaved RA codes 
IZAq(N) was originally derived in Section 5], and it is given by 



(N\(qN-h\( h-1 \ 

,NSRA(7v, g ) _ UAL^jAr^l-iJ 

\qw) 



and therefore the distance spectrum of the ensemble is given by 



miii(JV,L^J) (N\(qN-h\( h-1 \ 

h lqN\ 
w = l \qw) 



<h<qN 



A^ SRA = for 1 < h < [|"| , and ^4^ SRA ( Ar,,? ) — \ since the all-zero vector is always a codeword 
of a linear code. The asymptotic exponent of the distance spectrum of this ensemble is given by 

There are (qN)\ ways to place qN bits. However, permuting the q repetitions of any of the N information bits 
does not affect the result of the interleaving, so there are |^jv possible ways for the interleaving. Strictly speaking, 
by permuting the iV information bits, the vector space of the code does not change, which then yields that there are 
(q0"iv! distinct RA codes of dimension k and number of repetitions q. 
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(see [IT!) 

r [NSRA(g)]/o A lim r [NSRA(AT,q)] / « 
N— »oo 

max (- fl - H(u) + (1 - <5)ff ( - ) +8h(£)\. (17) 

0<«<min(2<5,2-2<5) ^ \ Q J ~ J \25J) 



o 
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Figure 3: Plot of the normalized asymptotic distance spectra for the ensembles of fully random 
block codes and uniformly interleaved and non-systematic repeat-accumulate (NSRA) codes of 
rate g bits per channel use. The curves are depicted as a function of the normalized Hamming 
weight (5), and their calculations are based on ((To^ and Q17|). 

The IOWEs and distance spectra of various ensembles of irregular repeat-accumulate (IRA) 
and accumulate-repeat-accumulate (ARA) codes are derived in [H I16j. 



2.4 The DS2 Bound for a Single MBIOS Channel 

The bounding technique of Duman and Salehi originates from the 1965 Gallager bound ^S] 
which states that the conditional ML decoding error probability P e \ m given that a codeword x m 
(of block length n) is transmitted is upper-bounded by 

where p n {y\x) designates the conditional pdf of the communication channel to obtain an n-length 
sequence y at the channel output, given the n-length input sequence x. 

Unfortunately, this upper bound is not calculable in terms of the distance spectrum of the code 
ensemble, except for the particular cases of ensembles of fully random block codes and orthogonal 
codes transmitted over a memoryless channel, and the special case where p = 1, A = 0.5 in which the 
bound reduces to the union-Bhattacharyya bound. With the intention of alleviating the difficulty 
of calculating the bound for specific codes and ensembles, we introduce the function ^^(y) which 
is an arbitrary probability tilting measure. This function may depend in general on the index 
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m of the transmitted codeword |32| . and is a non- negative function which satisfies the equality 
f \l4"^(y) dy = 1. The upper bound in (|18|) can be rewritten in the following equivalent form: 

Pe\m <E^ n \y_) Urr\y_r"pn [y_\^) £ (j^^) ^ > 0- (19) 

Recalling that Vl/n is a probability measure, we invoke Jensen's inequality in (|19j) which gives 



which is the DS2 bound. This expression can be simplified (see, e.g., [22]) for the case of a single 
memoryless channel where 



n 



Pn(y\x) = Y[p(Vi 
i=l 

Let us consider probability tilting measures ^ { ™\y) which can be factorized into the form 

n 

i=i 

recalling that the function ip( m ) may depend on the transmitted codeword x m . In this case, the 
bound in (|20|) is calculable in terms of the distance spectrum of the code, thus not requiring the 
fine details of the code structure. 

Let C be a binary linear block code whose block length is n, and let its distance spectrum be 
given by { A/j}^ =0 . Consider the case where the transmission takes place over an MBIOS channel. 
By partitioning the code into subcodes of constant Hamming weight, let Ch be the set which includes 
all the codewords of C with Hamming weight h and the all-zero codeword. Note that this forms a 
partitioning of a linear code into subcodes which are in general non-linear. We apply the DS2 bound 
on the conditional ML decoding error probability (given the all- zero codeword is transmitted), and 
finally use the union bound w.r.t. the subcodes {Ch} in order to obtain an upper bound on the ML 
decoding error probability of the code C. Referring to the constant Hamming weight subcode Ch, 
the bound (|2fl|) gives 

p.ioW< w p |^E^) 1_ 'P(y|o)^ {^m'-My^pm^ J ^So 1 - (21 

Clearly, for an MBIOS channel with continuous output, the sums in (1211) are replaced by integrals. 
In order to obtain the tightest bound within this form, the probability tilting measure tp and the 
parameters A and p are optimized. The optimization of ip is based on calculus of variations, and is 
independent of the distance spectrum. 

Due to the symmetry of the channel and the linearity of the code C, the decoding error probabil- 
ity of C is independent of the transmitted codeword. Since the code C is the union of the subcodes 
{Ch}, the union bound provides an upper bound on the ML decoding error probability of C which 
is expressed as the sum of the conditional decoding error probabilities of the subcodes Ch given 
that the all-zero codeword is transmitted. Let d m i n be the minimum distance of the code C, and R 
be the rate of the code C. Based on the linearity of the code, the geometry of the Voronoi regions 
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implies that one can ignore those subcodes whose Hamming weights are above n(l — R) (see (2j). 
Hence, the expurgated union bound gets the form 

n{l-R) 

Pc< Y, P e|0W- (22) 

h — drain 

For the bit error probability, one may partition a binary linear block code C into subcodes w.r.t. 
the Hamming weights of the information bits and the coded bits. Let C w ^ designate the subcode 
which includes the all-zero codeword and all the codeowrds of C whose Hamming weight is h and 
their information bits have Hamming weight w. An upper bound on the bit error probability of the 
code C is performed by calculating the DS2 upper bound on the conditional bit error probability 
for each subcode C w ^ (given that the all-zero codeword is transmitted), and applying the union 
bound over all these subcodes. Note that the number of these subcodes is at most quadratic in 
the block length of the code, so taking the union bound w.r.t. these subcodes does not affect the 
asymptotic tightness of the overall bound. Let {^L^} designate the IOWE of the code C whose 
block length and dimension are equal to n and k, respectively. The conditional DS2 bound on the 
bit error probability was demonstrated in |28l I29| to be identical to the DS2 bound on the block 
error probability, except that the distance spectrum of the code 

k 

A h = Y J A w , h , h = 0,...,n (23) 

w=0 

appearing in the RHS of (|2Tj) is replaced by 

k 

A 'h = Y,{j) A ^ h = 0,...,n. (24) 

w=0 

Since A' h < Ah then, as expected, the upper bound on the bit error probability is smaller than the 
upper bound on the block error probability. 

Finally, note that the DS2 bound is also applicable to ensembles of linear codes. To this end, 
one simply needs to replace the distance spectrum or the IOWE of a code by the average quantities 
over this ensemble. This follows easily by invoking Jensen's inequality to the RHS of (|21|) which 
yields that E[(A h )f] < (E[A h ])P for < p < 1. 

The DS2 bound for a single channel is discussed in further details in [9} I28[ 132] and the tutorial 
paper [23 Chapter 4] . 



11 



3 The Generalized DS2 Bound for Independent Parallel Channels 



In this section, we generalize the DS2 bound to independent parallel channels, and optimize the 
related probability tilting measures which enable to obtain the tightest bound within this form. 



3.1 Derivation of the New Bound 

Let us assume that the communication takes place over J statistically independent parallel channels 
where each one of the individual channels is a memoryless binary-input output-symmetric (MBIOS) 
with antipodal signaling, i.e., p(y\x = 1) = p(— y\x = — 1). We start our discussion by considering 
the case of a specific channel assignment. By assuming that all J channels are independent and 
MBIOS, we may factor the transition probability as 

J 

Pn{y\x m ) = H rj p(yik (m) ;i) (25) 

3=1 idl(j) 

which we can plug into (|2Uj) to get a DS2 bound suitable for the case of parallel channels. In order 
to get a bound which depends on one-dimensional sums (or one-dimensional integrals), we impose 
a restriction on the tilting measure l4 m) (-) in dUJ) so that it can be expressed J-fold product 
of one-dimensional probability tilting measures, i.e., 

J 

^ m) (y) = II II ^ m) (.yi-J)- (26) 

3=1 tezy) 

Considering a binary linear block code C, the conditional decoding error probability does not depend 
on the transmitted codeword, so P e = jj ^m=0 ^e\m = ^elo where w.o.l.o.g., one can assume that 
the all-zero vector is the transmitted codeword. 

The channel mapper for the J independent parallel channels is assumed to transmit the bits 
whose indices are included in the subset T(j) over the j-th channel where the subsets 
constitute a disjoint partitioning of the set of indices {1,2, ... ,n}. 

Following the notation in [2^) let ^-h\,h2,—,hj designate the split weight enumerator of the binary 
linear block code, defined as the number of codewords of Hamming weight hj within the J disjoint 
subsets for j = 1 . . . J. By substituting (|23|) and (J2H|) in (J2T?|) . we obtain 



c 



"(1)1 \1(J)\ J : 1 

L =0 hj=0 V j=li€I(j) 

L =o hj=o j=i ie xu) Vi 



p{yi\xj\j) 
p{yi\0;j) 

p(yi\xj-,j) 
p{yi\0;j) 



\i{J)\ j / ^ h 



E ^.■■aIIIE^^ 1 lp p(y\0;j) pP p(y\hj? 

hj=Q 3=1 \ V / 

II E^jJ^'PCvlOjj) 7 \ , U -£- \ (27) 



3=1 \ y 
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We note that the bound in (|27[) is valid for a specific assignment of bits to the parallel channels. 
For structured codes or ensembles, the split weight enumerator is in general not available when 
considering specific assignments. As a result of this, we continue the derivation of the bound by 
using the random assignment approach. Let us designate rij = \T(j)\ to be the cardinality of the set 
so E[rij] = ctjn is the expected number of bits assigned to channel no. j (where j = 1, 2, . . . , J). 
Averaging (|27[) with respect to all possible channel assignments, we get the following bound on the 
average ML decoding error probability: 

{n x nj J / i i a \ hj 

£•••£ A hlM _ hj H £v(y;i) 1_ ^(y|0;i)^(2/|i;i) A 
hi=0 hj=0 j=l V y J 

J I \ n 3~ h i ^ P 

i=i V j/ 

{ni raj J / 

£ ... A hlM _ hj H £^(y;i) 1_ ^(2/|0;j)^(2/|l;i) ; 
/i 1= hj=0 j=l \ 3/ / 

Ej "j=« 

J / \ n i~ h o \ P 

Jl\^2^(y-,j) l ~^p(y\0;jf p J > iVCa) (28) 

where -P/v(n) designates the probability mass function of the discrete random vector N_ = (ni, . . . , nj) 
After applying Jensen's inequality to the RHS of (|28|) and changing the order of summation, we 
get 



Pc ^ { £ £ £ A hlMi .„ >hj PN(n) 



rij>0 h=0 hi<ni,...,hj<nj 
J2nj=n h 1 + ...+h J =h 

.1 



E^^') 1 k 'p(v\Q\j)~^pW-\i) x ) 

3=1 V v / 

II £vW-^|0;i)M , ^S^ 1 • (29) 



Let the vector if = (/ii, . . . , hj) be the vector of partial Hamming weights referring to the bits 
transmitted over each channel (rij bits are transmitted over channel no. j, so < hj < rij). Clearly, 
S/=i fy? = ^ is the overall Hamming weight of a codeword in C. Due to the random assignment of 
the code bits to the parallel channels, we get 



n 

m,ra 2 , ... ,nj 



P N_{n) = L ) ot^ot^ ...a n / 



I h W n-h \ 

p /LL\ _ Vfei,...,fej/ Ui-fe b . ..,nj-hj) 

Ph\n{!1\W ~ 7 n \ 

\ni,...,nj) 

A hl ,h2,...,hjPN{n) 

= A h P mK {h\n) P K (n) 



h \ ( n — h 

hi,..., hjj \m - hi, . . . , nj - hj 



A h a^a¥...* n /[ u , _)(_ , , ) (30) 
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and the substitution of (|3U|) in $Z§§ gives 



p < * E E 



nj>0 h=0 h\<n\,...,hj<nj 
E rij-n 



h 

hi, h 2 , ■ ■ ■ , hj 



h\+...+hj=h 

n — h 

ni - hi,n 2 - h 2 , . . . ,nj - hj 



nf ^ — > 1—— 1 — Ap 

[ oij ^ip{y;j) "p{y\0;j) ? p(y\l;j) 

j=i V y 

II ( a i ^2!^(y'^) x ~~ p p{y\^j)~ p 



rij-hj \ p 



j=i \ y / ) 

Let kj — rij — hj for j = 1, 2, . . . , J, then by changing the order of summation in the above bound, 
we obtain 



Pe < < 



5>* E 

/i=0 /ii,...,hj>0 
/ii+...+hj=/i 



/l 

/ii, /i 2 , . . . ,hj 



ni v — -\ 1— 1 1 — Ap 



E 



fci,...,fc./>0 
fci+...+fcj=n— /i 



n — h 
ki,k 2 ,...,kj 



j'=i 



Since X)f=i fy? = ^ an d X)f=i kj = n — h, the use of the multinomial formula gives 



Pe < < 



E A I E q j E^^'- 7 ') 1 ? p(2/l ;i) p P p(y|!;i) 



h=0 



-h V" o < p < 1 
A > 

j = i ... j 



(31) 



which forms a generalization of the DS2 bound for independent parallel channels, where the bound 
is averaged over all possible channel assignments. This result can be applied to specific codes as 
well as to structured ensembles for which the average distance spectrum Ah is known. In this case, 
the average ML decoding error probability P e is obtained by replacing in Q31JI with A^. 2 

In the continuation of this section, we propose an equivalent version of the generalized DS2 
bound for parallel channels where this equivalence follows the lines in |29[ I32j . Rather than relying 
on a probability (i.e., normalized) tilting measure, the bound will be expressed in terms of an un- 
normalized tilting measure which is an arbitrary non-negative function. This version will be helpful 
later for the discussion on the connection between the DS2 bound and the 1961 Gallager bound 
for parallel channels, and also for the derivation of some particular cases of the DS2 bound. We 



2 This can be shown by noting that the function f(t) — t p is convex for < p < 1 and by invoking Jensen's 
inequality in 1311 . 
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begin by expressing the DS2 bound using the un-normalized tilting measure Cri"^ which is related 

Gt ] (y) P n(y\x m ) 



to by 



Y, G t\i)Pn{i\3f' 



(32) 



Substituting (jHU) in (J2UJ) gives 



l-P 



Pn(y\x r 
p n (y\x n 



w p 



< p < 1 
A > 



\ y j ym'^m y 

As before, we assume that G^ m) can be factored in the product form 

J 

G ( ™\y) = H II gfaj). 

3=1 

Following the algebraic steps in (|2*T)) - (|31j) and averaging as before also over all the codebooks of the 
ensemble, we obtain the following upper bound on the ML decoding error probability: 



j=i V y 



Pe = P c \0 < 




i-p - 



^2 a J [^aiyJ) 1 p p(y\0;j) 



j=i \ y 



l—p n—h ^ P 

p 



< p < 1 
A > 



(33) 



Note that the generalized DS2 bound as derived in this subsection is applied to the whole code 
(i.e., the optimization of the tilting measures refers to the whole code and is performed only once 
for each of the J channels). In the next subsection, we consider the partitioning of the code to 
constant Hamming weight subcodes, and then apply the union bound. For every such subcode, we 
rely on the conditional DS2 bound (given the all- zero codeword is transmitted), and optimize the 
J tilting measures separately. The total number of subcodes does not exceed the block length of 
the code (or ensemble), and hence the use of the union bound in this case does not degrade the 
related error exponent of the overall bound, but on the other hand, the optimized tilting measures 
are tailored for each of the constant-Hamming weight subcodes, a process which can only improve 
the exponential behavior of the resulting bound. 



3.2 Optimization of the Tilting Measures for the Generalized DS2 Bound 

In this section, we find optimized tilting measures {V'(') j)}/=i which minimize the DS2 bound (|31|) . 
The following calculation generalizes the analysis in 32 for a single channel to the considered case 
of an arbitrary number ( J) of independent parallel MBIOS channels. 

Let C be a binary linear block code of length n. Following the derivation in |21M32j . we partition 
the code C to constant Hamming weight subcodes {Ch}^ =0 , where Ch includes all the codewords 
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of weight h (h = 0, ...,n) as well as the all-zero codeword. Let P e \o(h) denote the conditional 
block error probability of the subcode Ch under ML decoding, given that the all-zero codeword is 
transmitted. Based on the union bound, we get 



P e <X>e|oW- (34) 



h=0 



As the code C is linear, P e \o(h) = for h = 0, 1, . . . ,d m i n — 1 where d m \ n denotes the minimum 
distance of the code C. The generalization of the DS2 bound in (|3*Tj) gives the following upper 
bound on the conditional error probability of the subcode Ch'- 



p e \o(h) < (A h yi 



( , J 

1=A£ - \ 



2_^aj2_^ip(y;j) "p(y\0;j) p p(y\l;j) y 



l-8 \ n P 

h 



^aj^^yj) 1 pp(y\0;j)p j }> , < p < 1, A > 0, S 4 -. (35) 

Note that in this case, the set of probability tilting measures {^("i j)}/=i may also depend on the 
Hamming weight (/i) of the subcode (or equivalently on 5). This is the result of performing the 
optimization on every individual constant-Hamming subcode instead of the whole code. 

This generalization of the DS2 bound can be written equivalently in the exponential form 

P e \o(h) < e-^f^WKD <p<l, A>0, 6 4 -. (36) 
1 n 

where 



Ef S2 (\,p,j,{ aj }) 4 - P r c (5) - P 5 in I ^^^(y-J^^Piylo-J^Piyli-J) 

-Kl-^ln (^ai^^i) 1 "^^)' ] (37) 
\i=i y / 

and r c (<5) designates the normalized exponent of the distance spectrum as in (|14|h 
Let 

5i(y;i)=Ky|0;j)", 52(2/; j) =p(y|0;j> (Sjy) ( 38 ) 

then, for a given pair of A and p (where A > and < p < 1), we need to minimize 

5\n I ^a^^v^y; i) 1 "^?/; j) J + (1-5) in ^aj^V(y;j) 1_ ^fl r i(y;. 

over the set of non-negative functions i/>(- ;j) satisfying the constraints 

S>(y;i) = i, j = i.-.J. (39) 
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To this end, calculus of variations provides the following set of equations: 

a 3 {l-5){l-l) gi {y;j) 



V>(y; j) 



+ & = 0, j = 1, . . . , J 



j l— - 

E y Ei=i a i^(y; i) p s-2(y;j 

where £j is a Lagrange multiplier. The solution of (|4()|) is given in the following implicit form: 

V>(y; j) = (h,j9i(y;j) + k 2 ,jg2{y;j)) p , feij, foj > o, j = i,...,J 

where 

^ a j^(y>j) ' 7 9i(y,j) 



(40) 



5 3 -=i j,ey 



1-5 J _ x 

3=1 y&y 



We note that k = ^4 in the RHS of (|41jl is independent of j. Thus, the substitution /3j = k^- 
gives that the optimal tilting measures can be expressed as 

V>(y;j) = /3j[gi(y,j) +kg 2 (y,j)) p 



(41) 



= PjP(y\0;j) 

By plugging into (|1T|) we obtain 



l + k 



p(y\i;j) 
p{y\0;j) 



y^y j = l, 



J. 



(42) 



J 

EE' 




l + k 


f p{y\i;j)\ x 
\p(y\0;j)J _ 


1 


EE W 


p i m ^ ( v{y\^\i)\ x 

pm3, {pmf)) 


Vp(y|0;i)y _ 


> 



(43) 



and from (|38|) and (|39j) . /3j which is the appropriate factor normalizing the probability tilting 
measure tp(-;j) in fl2|) is given by 



A- 



p(y|i;iy 
p(y|0; j) 



j = 1, . . . , J. 



(44) 



Note that the implicit equation for k in (|43j) and the normalization coefficients in 1)44(1 generalize 
the results derived in |28l Appendix A] (where k is replaced there by a). The key result here is 
that the values of in (flTj) are independent of j (where j £ {1,2,... , J}), a property which 
significantly simplifies the optimization process of the J tilting measures, and leads to the result in 

(El. 

For the numerical calculation of the bound in (|35|) as a function of the normalized Hamming 
weight 5 = ^, and for a fixed pair of A and p (where A > and < p < 1), we find the optimized 
tilting measures in (|42|) by first assuming an initial vector = . . . ,/3j) and then iterating 
between (|43(l and Q44j) until we get a fixed point for these equations. For a fixed 5, we need to 
optimize numerically the bound in (|36|) w.r.t. the two parameters A and p. 
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3.3 Statement of the Main Result Derived in Section 03 

The analysis in this section leads to the following theorem: 

Theorem 1 (Generalized DS2 bound for independent parallel MBIOS channels). Con- 
sider the transmission of binary linear block codes (or ensembles) over a set of J independent 
parallel MBIOS channels. Let the pdf of the j th MBIOS channel be given by p(-\0;j) where due 
to the symmetry of the binary-input channels p(y\0;j) = p(—y\l;j). Assume that the coded bits 
are randomly and independently assigned to these channels, where each bit is transmitted over one 
of the J MBIOS channels. Let a>j be the a-priori probability of transmitting a bit over the j th 
channel (j = 1,2,..., J), so that aj > and Ylj=i a j = 1- Then, the generalization of the DS2 
bound in (|3~Tj) provides an upper bound on the ML decoding error probability when the bound is 
taken over the whole code. By partitioning the code into constant Hamming-weight subcodes, l)35JI 
forms an upper bound on the conditional ML decoding error probability for each of these subcodes, 
given that the all-zero codeword is transmitted, and l)34jl forms an upper bound on the block error 
probability of the whole code (or ensemble). For an arbitrary constant Hamming weight subcode, 
the optimized set of probability tilting measures {ip(-;j)}j =i which attains the minimal value of 
the conditional upper bound in (|33|) is given by the set of equations in (J4"2"|) - (|4"H) . 

4 Generalization of the 1961 Gallager Bound for Parallel Channels 
and Its Connection to the Generalized DS2 Bound 

The 1961 Gallager bound for a single MBIOS channel was derived in ^21; an d the generalization of 
the bound for parallel MBIOS channels was derived by Liu et al. |2l]|. In the following, we outline 
its derivation in [2J which serves as a preliminary step towards the discussion of its relation to the 
generalized DS2 bound from Section 13 In this section, we optimize the probability tilting measures 
which are related to the 1961 Gallager bound for J independent parallel channels in order to get 
the tightest bound within this form (hence, the optimization is carried w.r.t. J probability tilting 
measures). This optimization differs from the discussion in [2^ where the authors choose some 
simple and sub-optimal tilting measures. By doing so, the authors in derive bounds which are 
easier for numerical calculation, but the tightness of these bounds is loosened as compared to the 
improved bound which is combined with the J optimized tilting measures (this will be exemplified 
in Section for turbo-like ensembles). 

4.1 Presentation of the Bound |21j 

Consider a binary linear block code C. Let x m be the transmitted codeword and define the tilted 
ML metric 



As noted in [221, D m {-, ■) is in general not computable at the receiver. It is used here as a conceptual 
tool to evaluate the upper bound on the ML decoding error probability. The received set y n is 




(45) 
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expressed as a union of two disjoint subsets 

y$ = {y£y n :D m (x m ,y)<nd} 

y% = {y^y n --D m (x m ,y)>nd} 

where d is an arbitrary real number. The conditional ML decoding error probability can be ex- 
pressed as the sum of two terms 

P e | m = Prob(error,y G y£) + Prob(error,y G y£) 

which is upper bounded by 

P e \ m < Prob(y G + Prob(error, y G 3%) • (46) 

We use separate bounding techniques for the two terms in (|46|). Applying the Chernoff bound on 
the first term gives 

Pi 4 Prob(y a b ")<E {e sW ) , s>0 (47) 

where 

W A ln (j^M\_ nd _ (48) 

Using a combination of the union and Chernoff bounds for the second term in the RHS of (|4fij) 
gives 

P 2 = Prob(error,y G yg) 

= Prob ^Z) m (x m ' ,y) < D m (x m ,y) for some m ^ m, y&y'g 

< Prob ( D m{x m \y) < D m (x m ,y), D m (x m ,y) < nd) 

< E (exp(tU m t + rW)) , t > 0, r < (49) 

where, based on (f4*5|) . 

L7 m , = L> m fe m ,y) - D m (x m ',y) = In r^ffj. I ■ ( 50 ) 

Consider a codeword of a binary linear block code C which is transmitted over J parallel MBIOS 
channels. Since the conditional error probability under ML decoding does not depend on the 
transmitted codeword, one can assume without loss of generality that the all-zero codeword is 
transmitted. As before, we impose on the function fn"'\y) the restriction that it can be expressed 
in the product form 

J 

ftHy) = U II f(yi;j)- (5i) 

3=1 iex(J) 
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For the continuation of the derivation, it is assumed that the functions /(■; j) are even, i.e., f(y;j) 
f{-y;j) for all y G y. Plugging (j5U|) and (JHU into (gl|) and (J^J) we get 



II E^l ^') 1_s /(y;i) 



" 1 v e~ nsd s > 



-nrd 



n\ nj 

E-E 

h, 1= 7ij=0 



E**^4«y 



n 

i=i 



E^i°;-?) 1_r /(^T 



e- nrd , t,r<0 



(52) 



(53) 



where as before, we use the notation nj = \T(j)\. Optimizing the parameter t gives the value in 
[121 Eq. (3.27)] 

r — 1 

t = V~ ■ (54) 



Let us define 



G(r;j) 4 ^p(y|0;j) 1 -7(2/;i) r 

^(r;i) = ^ b(z/|0; j>(z/|i; i)] 1 ^ /(y; i) r 



(55) 
(56) 



Substituting ((5*1)) into (""""")) ■ combining the bounds on Pi and P2 in (|52j) and "5*""|) . and finally 
averaging over all possible channel assignments, we obtain 



P c < E 



E E A hl _ hj l{[Z(r;j)p[G(r;j)r^e- 
3=1 

E fe j =/i 



■nrd 



i=i 



E 

rij>0 

E ™j= n 



E E A hl _ hj Y[[Z(r;j)p[G(r;j)r^e- m 

h=10<hj<rij j=l 
E hj— 7i 

}r < 
Pjv(n) , s>0 
— 00 < <i < 00 



nrc( 



(57) 
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Following the same procedure for random assignments as in (|3Up and (|31|). we obtain 



n—h ' 



h=l \j=l 



J =1 



-nrd 



+ ^2 aj G(s;j)j e~ nsd . 
Finally, we optimize the bound in (|58|) over the parameter d which gives 



(58) 



R < 2 h ^ < 



E^ 

h=l 



3=1 



^2 aj G(r;j) 
3=1 



-h \ P , j >> n(l-p) 



i=i 



(59) 



where r < 0, s > 0, and 



s — r 



< p < 1 . 



(60) 



The bound in (|59]1 . originally derived in |21j . forms a natural generalization of the 1961 Gallager 
bound for parallel channels. 



4.2 Connection to the Generalization of the DS2 Bound for Independent and 
Memoryless Parallel Channels 

Divsalar ||, and Sason and Shamai |29] 132] discussed the connection between the DS2 bound and 
the 1961 Gallager bound for a single MBIOS channel. In this case, it was demonstrated that the 
former bound is necessarily tighter than the latter bound. At a first glance, one would expect this 
conclusion to be valid also for the case where the communication takes place over J independent 
parallel MBIOS channels (where J > 1). We show in this section the difficulty in generalizing 
this conclusion to an arbitrary number of independent parallel channels, and the numerical results 
presented in Section Q support the conclusion that the DS2 bound is not necessarily tighter than the 
1961 Gallager bound when considering communications over an arbitrary number of independent 
parallel channels. 

In what follows, we will see how a variation in the derivation of the Gallager bound leads to a 
form of the DS2 bound, up to a factor which varies between 1 and 2. To this end, we start from 
the point in the last section where the combination of the bounds in (]52j) and (|53|) is obtained. 
Rather than continuing as in the last section, we first optimize over the parameter d in the sum of 
the bounds on P\ and P2 in and (|53*|) . yielding that 



P e < 2^) ^ A hl _ h] l[V(r,t,jpG(r;j) 



v h=l hi,...,hj 
Ej hj=h 



3=1 
J 



P J 

> \{G(s-3) n ^- p) 

3=1 



2 h (p) 



C n 

E E A hl _ hj \[\v{r,t-j)G(s;jp 

\h=l hi,...,hj 3=1 
J 

II [G(r;j)G(s; 



3) " 



, t,r < 0, s > 
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where 



nn*;j) = EKy|0;j) 1 -7(y;. 



(61) 



G(-;j) is introduced in (|55[) for j = 1, . . . , J, and p is given in ()6(JI) . Averaging the bound with 
respect to all possible channel assignments, we get for < p < 1 



E 



nj>0 
. n 3 =; 



n j 



h=l h\,...,hj 

E,- ' l j= h 



II [G(r;i)G( S ; 



i=i 



^iv(n) 



< 2 /l ^ 



E E E A hl) ... >h .P K (n)]j[v(r,t-,j)G(s;j) 1 -^ 



h j 



n,>0 h=l hi,...,hj 



3=1 



J 

H[G(r;j)G(s;j) 



i-p 



(62) 



where we invoked Jensen's inequality in the last step. Following the same steps as in ((2*5)) -(|3*T j) . we 

get 



p < 2 h( P ) 



E^ h E a /( r '^') G ( s ^' 



1-p 



n—h 



^ayG(r;j)G(s;.7j p 



(63) 



where from (|55|). (|60|) and (ffi"TJ) 

c^^^E^i ^': 

y 

G(r;j) = ^p(y|0; 
V{r,t;j) = E^l ^ 



f(y;j) 
p(y\0;j] 



f(y;j) V (1 ~*> 



p(y|0;i 
/(y; j) V (1 ^ } ( p(y\°'J) 



p(y\0;j) 



p(y\i;j) 



(64) 



Setting A = — t, and substituting in (|64|) the following relation between the Gallager's tilting 
measures and the un-normalized tilting measures in the DS2 bound 



g(y;j) 



f(y;j) 
p{y\0;j) 



, j = l,2,...,J 



(65) 
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we obtain 




J>j l^givJ) 1 ^p(2/|0;i) 1 - A p(y|l;i) A ) 

l-p -i h r 

, < p < 1 



(66) 



which coincides with the form of the DS2 bound given in (1331) (up to the factor 2 ft W which lies 
between 1 and 2), for those un-normalized tilting measures <?(•; j) such that the resulting functions 
/(si) i n <E3) are even. 

Discussion. The derivation of the 1961 Gallager bound first involves the averaging of the bound 
in (|57j) over all possible channel assignments and then the optimization over the parameter d in 
(|58jl. To show a connection to the DS2 bound, we had first optimized over d and then obtained the 
bound averaged over all possible channel assignments. The difference between the two approaches 
is that in the latter, Jensen's inequality had to be used in (|6*2*|) to continue the derivation (because 
the expectation over all possible channel assignments was performed on an expression raised to the 
p-th power) which resulted in the DS2 bound, whereas in the derivation of the need for Jensen's 
inequality was circumvented due to the linearity of the expression in (|57|). We note that Jensen's 
inequality was also used for the direct derivation of the DS2 bound in (|3Tj) . In the particular case 
where J = 1, there is no need to apply Jensen's inequality in (|28|) and (|62|) . By particularizing 
the model of parallel channels to a single MBIOS channel, the DS2 bound is tighter than the 1961 
Gallager bound (as noticed in [35]) due to the following reasons: 



• For the 1961 Gallager bound, it is required that f(-;j) be even. This requirement inhibits the 
optimization of in Section 13.21 because the optimal choice of ip(-;j) given in 1)42(1 leads to 
functions f(-;j) which are not even. The exact form of f(-;j) which stems from the optimal choice 
of ip(-;j) is detailed in Appendix lA.il 

• The absence of the factor 2 h( - p ^ (which is greater than 1) in the DS2 bound implies its superiority. 
Naturally, this factor is of minor importance since we are primarily interested in the exponential 
tightness of these bounds. 



For the case where J > 1, it is not clear from the discussion so far which of the bounds is tighter. 
We note that, as in the case of J = 1, the optimization over the DS2 tilting measure is over a larger 
set of functions as compared to the 1961 Gallager tilting measure; hence, the derivation of the DS2 
bound from the Gallager bound only gives an expression of the same form and not the same upper 
bound (disregarding the 2 h ^ p ' constant). We will later compare the optimized DS2 bound and the 
optimized 1961 Gallager bound. 

Although we have shown here how to obtain a form of the DS2 bound from the Gallager bound 
by invoking Jensen's inequality, it is insightful to use the relations connecting the DS2 bound with 
the Gallager bound in the derivation above without invoking Jensen's inequality, in order to obtain 
from the 1961 Gallager bound a tighter bound than the one in (|66|) . This is possible if we start 
from the bounds in (|52fl and (|53|) , optimize over all possible channel assignments and then optimize 
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over d. In this way the use of Jensen's inequality is avoided, and one can obtain the following upper 
bound 



Pe < 2 h ^\Y j A h 
\h=0 



^2 a i^2 9(y, j)p(y\0; j 
vi=i y 



E a i 1^2 giv,]) 1 "piyl&J) 1 x p(v\i;j) 

j=i V y 

l-p -i h 

E a J ( ^2,9(y\j) l ~~ p p{y\^\j) 

i=i V y / 

l— p -i n—h \ P 



E"j ^ 9(y, j)p{y\0; j) 



(67) 



Some technical details related to the derivation of this bound are provided in Appendix IA.2I This 
upper bound is clearly tighter than the one in (J66|). The reader should note, however, that these 
two bounds coincide whenever the expression ^ y g{y\ j)p{y\ty j) is independent of j. 



4.3 Optimized Tilting Measures for the Generalized 1961 Gallager Bound 

We derive in this section optimized tilting measures for the 1961 Gallager bound. These optimized 
tilting measures are derived for random coding, and for the case of constant Hamming weight codes. 
The 1961 Gallager bound will be used later in conjunction with these optimized tilting measures in 
order to get an upper bound on the decoding error probability of an arbitrary binary linear block 
code. To this end, such a code is partitioned to constant Hamming weight subcodes (where each 
one of them also includes the all-zero codeword), and a union bound is used in conjunction with the 
calculation of the conditional error probability of each subcode, given that the all-zero codeword is 
transmitted. Using these optimized tilting measures improves the tightness of the resulting bound, 
as exemplified in the continuation of this paper. 



4.3.1 Tilting Measures for Random Codes 

Consider the ensemble of fully random binary block codes of length n. Substituting the appropriate 
weight enumerator (given in p5|)) into ([58)1 . we get 



p < 2 -n(l-R) 



\^2 a j^2 [pivl '^) 1 ^ la " f(y 



j=l y 



, 3 y V, e -nrd 



+ < 



nsd 



j=l y 



r < 

s > (68) 
-co < d < +co 



where we rely on (|55|) and 1)56(1. use the symmetry of the channels and the fact that we require the 
functions f(-;j) (j = 1, . . . , J) to be even. To optimize ({55} over all possible tilting measures, we 
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apply calculus of variations. This procedure gives the following equation: 

J _ 2 

i=i 
J 

-L^a j (p(y|0;j) 1 - s +p(y|l;j) 1 - s ) 2 /(y;ir 1 = Vy. 
i=i 

where LsR. This equation is satisfied for functions which are given in the form 



f{y;j) = K < 



p(y\0;j) ^ +p(yl 1 u') 2 7 



p(y|0; j) 1 s +p(y\l;j] 



l-s 



K G 



(69) 



This forms a natural generalization of the tilting measure given in |12| Eq. (3.41)] for a single 
MBIOS channel. We note that the scaling factor K may be omitted as it cancels out when we 
substitute JMJ) in (|59|) . 



4.3.2 Tilting Measures for Constant Hamming Weight Codes 



The distance spectrum of a constant Hamming weight code is given by 



1, if ti = 
A h , if h' = h 
0, otherwise 



(70) 



Substituting this into (|59j) and using the symmetry of the component channels and the fact that 
the tilting measures /(•; j) are required to be even, we get 

hp 

P c < 2^)^^a i ^[p(y|0;j>(y|l;j)] i ? : /(y;i) 
I i=i y 



EyEW^') 1 " +p(y\Uj) 1 - r ] /(vjjT 



i=i ?/ 
j 

a 



(n—h)p 



n(l-p) 



Ef £ [Pd/I0;j) 1_s + /(v;j) f 



i=i 3/ 



r < 0, s > 0, p 



s — r 



(71) 



Applying calculus of variations to (|71|) yields (see Appendix IA.3I for some additional details) that 
the following condition should be satisfied for all values of y G y~. 
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£«,{ W 0;i)1 " +P(y|l;i) 1_s ] f(y,j) s - r + K 1 b(y|0;i)p(y|l;i)] 
i=i 

+K 2 [p(y|0;i) 1 -'-+p(y|l;j) 1 - r ]} 
where K\, if 2 G R. This condition is satisfied if we require 



(72) 



[p(y\0;j) l - s + P (y\l;j) l - S ] f(y;j) s - r + K t [ P (y\0; j) P (y\i; j)] 
+K 2 [p(y\0;j) 1 - r +p(y\l;j) 1 - r }^0, 



1-7- 
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VyG^, i = l,...,J. 
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The optimized tilting measures can therefore be expressed in the form 



f(yj) 



( , ,„ . l-s(l-p L ) l-s(l- 

ci(p{y\0;j) 2 +p(y\l;j) 2 




+ 



p{y\0;jy- s +p{y\l;jy- s 
di (p(j/|0; j) 1 -** 1 -'" 1 ) + P (y\l;j) 1 ~ s{1 - p ~ 1) ) 



£ 



, ci.di 6l,s > 0,0 < p< 1 (73) 



p(y\0; j) 1 s +p(y\i;j) 



where we have used (|60j) . This form is identical to the optimal tilting measure for random codes if 
we set d\ = 0. It is possible to scale the parameters c\ and d\ without affecting the 1961 Gallager 
bound (i.e., the ratio ^ cancels out when we substitute (|73*)l in (|59|)). Furthermore, we note that 
regardless of the values of c\ and di, the resulting tilting measures are even functions, as required 
in the derivation of the 1961 Gallager bound. 

For the simplicity of the optimization, we wish to reduce the infinite intervals in (j73|) to finite 
ones. It is shown in |27l Appendix A] that the optimization of the parameter s can be reduced 
to the interval [0, 1] without loosening the tightness of the bound. Furthermore, the substitution 
c = 2c^+3di ' as suggested in j27J Appendix B], enables one to express the optimized tilting measure 
m using an equivalent form where the new parameter c ranges in the interval [0,1]. The 
numerical optimization of the bound in ()73|) is therefore taken over the range of parameters < 
< s < 1, < c < 1. Based on the calculations in 27, Appendices A, B], the functions 
/(•; j) get the equivalent form 



By reducing the optimization of the three parameters over the unit cube, the complexity of the 
numerical process is reduced to an acceptable level. 

4.4 Statement of the Main Result Derived in Section UJ 

The analysis in this section leads to the following theorem: 

Theorem 2 (Generalized 1961 Gallager bound for independent parallel MBIOS chan- 
nels with optimized tilting measures). Consider the transmission of binary linear block codes 
(or ensembles) over a set of J independent parallel MBIOS channels. Following the notation in 
Theorem 1, the generalization of the 1961 Gallager bound in ()59|) provides an upper bound on the 
ML decoding error probability when the bound is taken over the whole code (as originally derived 
in |U). By partitioning the code into constant Hamming- weight subcodes, the generalized 1961 
Gallager bound on the conditional ML decoding error probability of an arbitrary subcode (given 
the all- zero codeword is transmitted) is provided by (|71|) . and ()34|) forms an upper bound on the 
block error probability of the whole code (or ensemble). For an arbitrary constant Hamming weight 
subcode, the optimized set of non-negative and even functions {/(•; j)}/=i which attains the min- 
imal value of the conditional bound in (|71|). is given by (|74|): this set of functions is subject to a 
three-parameter optimization over a cube of unit length. 





(74) 
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5 Special Cases of the Generalized DS2 Bound for Independent 
Parallel Channels 



In this section, we rely on the generalized DS2 bound for independent parallel MBIOS channels, 
as presented in Section 13.11 and apply it in order to re-derive some of the bounds which were 
originally derived by Liu et al. |21| . The derivation in [2^ is based on the 1961 Gallager bound 
from Section [4. 11 and the authors choose particular and sub-optimal tilting measures in order to 
get closed form bounds (in contrast to the optimized tilting in Section 14.31 which lead to more 
complicated bounds in terms of their numerical computation). In this section, we follow the same 
approach in order to re-derive some of their bounds as particular cases of the generalized DS2 bound 
(i.e., we choose some particular tilting measures rather than the optimized ones in Section 2(1 . In 
some cases, we re-derive the bounds from [2j| as special cases of the generalized DS2 bound, or 
alternatively, obtain some modified bounds as compared to |21| . 

5.1 Union-Bhattacharyya Bound in Exponential Form 

As in the case of a single channel, it is a special case of both the DS2 and Gallager bounds. By 
substituting r = in the Gallager bound or p = 1, A = 0.5 in the DS2 bound, we get 



where 7 is given by (jHJ) and denotes the average Bhattacharyya parameter of J independent parallel 
channels. Note that this bound is given in exponential form, i.e., as in the single channel case, 
it doesn't use the exact expression for the pairwise error probability between two codewords of 
Hamming distance h. For the case of the AWGN, a tighter version which uses the Q-function to 
express the exact pairwise error probability is presented in Appendix C. 

5.2 The Sphere Bound for Parallel AWGN Channels 

The simplified sphere bound is an upper bound on the decoding error probability for the binary- 
input AWGN channel. In |21j . the authors have obtained a parallel-channel version of the sphere 
bound by making the substitution f(y;j) = -^j= in the 1961 Gallager bound. We will show that 
this version is also a special case of the parallel-channel DS2 bound. By using the relation (|65j) . 
between Gallager's tilting measure and the un-normalized DS2 tilting measure, we get 
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h=l 




so that 
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By introducing the two new parameters (3 = 1 — s (l — ij and A = (3/2 we get 



+oo 



g(y,j)p(y\0;j)dy 



— oo 
+oo 



1-p 



3(2/; j) 1 p p{y\0;j)dy = (3 * 

13 



— oo 
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Next, by plugging (f75|) into we get 



Pc < < 



/i=0 



1-^ 2 



l-(3p 



7i = e J - 



< p < 1 



(75) 



(76) 



This bound is identical to the parallel-channel simplified sphere bound in |21[ Eq. (24)], except 
that it provides a slight improvement due to the absence of the factor 2 h ^ which appears in |211 
Eq. (24)] (a factor bounded between 1 and 2). 

We observe that in (|75|) . j y g(y; j)p(y\0; j)dy is independent of j, so the fact that the expressions 
in (|76|) and |211 Eq. (24)] coincide is not surprising in light of the discussion at the end of Section^H 



5.3 Generalizations of the Shulman-Feder Bound for Parallel Channels 



In this sub-section, we present two generalizations of the Shulman and Feder (SF) bound, where 
both bounds apply to independent parallel channels. The first bound was previously obtained by 
Liu et al. [2J| as a special case of the generalization of the 1961 Gallager bound, and the second 
bound follows as a particular case of the generalized DS2 bound for independent parallel channels. 

By substituting in (|59*|) the tilting measure and the parameters (see Eq. (28)]) 



f{y;j) 



1 i 1 i 

-Ky|0;i) 1+ " + ^p(y\i;j) 1+p 



\_-p_ 
i + p' 



o < p < 1 



(77) 



straightforward calculations for MBIOS channels give the following bound which was originally 
introduced in |211 Lemma 2]: 



p < 2 h{p) 2 nRp m£JX 



A h 



l<h<n 2~ n ^- R ) (I) 



(78) 
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Considering the DS2 bound, it is possible to start from Eq. (|33|) and take the maximum distance 
spectrum term out of the sum. This gives the bound 



max 



<h<n 2-< 1 - R ){ n h ) 



3=1 
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p(y\i;j] 
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< p < 1. (79) 



28 



Using the J un-normalized tilting measures 



1 1 1 1 



p(y\0;j) 1+p , j = i,2,..., J 



(80) 



and setting A = in (|79|). gives the following bound due to the symmetry at the channel outputs: 

\ P 



Pe < 2 
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i<h<n 2""( 1 ~ i? ) (I) I 

^(y|0; j) 1 ^ + ^p(y\i;j)^» 



i \ np 



, < p < 1 



(81) 



which forms another possible generalization of the SF bound for independent parallel channels, 
where the latter variation follows from the generalized DS2 bound. Clearly, unless J = 1 (referring 
to the case of a single MBIOS channel), this bound is exponentially looser than the one in (|78|) . 
The fact that the bound in (|81|) is exponentially looser than the bound in (|78|) follows from the use 
of Jensen's inequality for the derivation of the generalized DS2 bound (see the move from (|28j) to 
(GHJ)). The tilting measures g(-;j) and f{-;j) (J = 1,...,J) in (|5U)) and (J77J), respectively, satisfy 
the relation in (|65ft . For the case where J = 1, these two bounds coincide (up to the 2 h ^ term 
which makes the 1961 Gallager bound at most twice larger than the DS2 bound), and one gets that 
this particular case of DS2 bound is identical to the SF bound, as originally observed in [32*] . 



5.4 Modified Shulman-Feder Bound for Independent Parallel Channels 

It is apparent from the form of the SF bound that its exponential tightness depends on the quantity 

x 2 -"M)(;) (82) 

which measures the maximal ratio of the distance spectrum of the considered binary linear block 
code (or ensemble) and the average distance spectrum of fully random block codes with the same 
rate and block length. One can observe from Fig. |3] (see p. |^J) that this ratio may be quite large 
for a non-negligible portion of the normalized Hamming weights, thus undermining the tightness of 
the SF bound. The idea of the modified Shulman-Feder (MSF) bound is to split the set of non-zero 
normalized Hamming weights ty n = {^,^,...,1} into two disjoint subsets and where the 
union bound is used for the codewords with normalized Hamming weights within the set and 
the SF bound is used for the remaining codewords. This concept was originally applied to the ML 
analysis of ensembles of LDPC codes by Miller and Burshtein [21]. Typically, the set consists 
of low and high Hamming weights, where the ratio in (|82j) between the distance spectra and the 
binomial distribution appear to be quite large for typical code ensembles of linear codes; the set 
is the complementary set which includes medium values of the normalized Hamming weight. 
The MSF bound for a given partitioning , is introduced in Lemma 3] , and gets the form 

P~e < Yl + 

2 H P ) 2 n Rp 2 _J_ h R){l) X J f> fc^l°;^ + \p{v\i;i)^\ P \ (83) 
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where 7 is introduced in ©, and < p < 1. Liu et al. prove that in the limit where the block length 
tends to infinity, the optimal partitioning of the set of non-zero normalized Hamming weights to 
two disjoint subsets ^~ and is given by (see |^| Eq. (42)]) 



Se 



n 



if - Slnj > H(S) + (I 
otherwise 



l)ln 2 



(84) 



where 



p(y\x;j) 



3=1 



x-e{-i,i} y 



l / 2 Y, x ie{-i,i}P{y\ x ';j) 



designates the average mutual information under the assumption of equiprobable binary inputs. 
Note that for finite block lengths, even with the same partitioning as above, the first term in the 
RHS of (j83j) can be tightened by replacing the Bhattacharyya bound with the exact expression for 
the average pairwise error probability between two codewords of Hamming distance h. Referring to 
parallel binary-input AWGN channels, the exact pairwise error probability is given in (IH3I) . thus 
providing the following tightened upper bound: 
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.(85) 



On the selection of a suitable partitioning of the set in (|85|) : The asymptotic partitioning 
suggested in (|84[) typically yields that the union bound is used for low and high values of normalized 
Hamming weights; for these values, the distance spectrum of ensembles of turbo-like codes deviates 
considerably from the binomial distribution (referring to the ensemble of fully random block codes 
of the same block length and rate). Let Si and S r be the smallest and largest normalized Hamming 
weights, respectively, referring to the range of values (S) in (|84*|) for which — <5j + -,..., 5 r }, 
and ^+ = ^, . . . , Si — ^} U {S r + ^, S r + ^, . . . , 1} be the complementary set of normalized 
Hamming weights. The subsets and ^~ refer to the discrete values of normalized Hamming 
weights for which the union bound in its exponential form is superior to the SF bound and vice versa, 
respectively (see (|83jl ) . Our numerical experiments show that for finite- length codes (especially, for 
codes of small and moderate block lengths) , this choice of Si and S r often happens to be sub-optimal 
in the sense of minimizing the overall upper bounds in (|83|) and (|85|) , This happens because for 
S = Si (which is the left endpoint of the interval for which the SF bound is calculated), the ratio of 
the average distance spectrum of the considered ensemble and the one which corresponds to fully 
random block codes is rather large, so the second term in the RHS of (|83|) and (|85|) corresponding 
to the contribution of the SF bound to the overall bound is considerably larger than the first term 
which refers to the union bound. Therefore, for finite-length codes, the following algorithm is 
proposed to optimize the partition ty n = U $~ : 



1. Select initial values Si and S ro (for Si and S r ) via (|84[) , If there are less than two solutions to the 
equation — (5 In 7 = H(S) + (I — 1) In 2, select ^+ = = cfi as the empty set. 

2. Optimize the value of Si by performing a linear search in the range [Si , 5 ro ] and finding the value 
of Si which minimizes the overall bound in the RHS of ()85|) . 
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This algorithm is applied to the calculation of the LMSF bound for finite-length codes (see, 
e.g., Fig. |4(b)| in p. !55|) . 



Clearly, an alternative version of the MSF bound can be obtained from the generalized DS2 
bound for parallel channels. In light of the discussion in Section 15.31 this version is also expected to 
be looser than the one in (|83j) . We address the MSF bound in Sectional where for various ensembles 
of turbo-like codes, its tightness is compared with that of the generalized DS2 and Gallager bounds. 

6 Inner Bounds on Attainable Channel Regions for Ensembles of 
Good Binary Linear Codes Transmitted over Parallel Channels 

In this section, we consider inner bounds on the attainable channel regions for ensembles of good 
binary linear codes (e.g., turbo- like codes) whose transmission takes place over independent parallel 
channels. The computation of these regions follows from the upper bounds on the ML decoding 
error probability we have obtained in Sections 01 and 0] (see Theorems ^ an d EJ , referring here to 
the asymptotic case where we let the block length tend to infinity. 

Let us consider an ensemble of binary linear codes, and assume that the codewords of each 
code are transmitted with equal probability. A J-tuple of transition probabilities characterizing a 
parallel channel is said to be an attainable channel point with respect to a code ensemble C if the 
average ML decoding error probability vanishes as we let the block length tend to infinity. The 
attainable channel region of an ensemble whose transmission takes place over parallel channels is 
defined as the closure of the set of attainable channel points. We will focus here on the case where 
each of the J independent parallel channels can be described by a single real parameter, i.e., the 
attainable channel region is a subset of M^; the boundary of the attainable region is called the 
noise boundary of the channel. Since the exact decoding error probability under ML decoding is in 
general unknown, then similarly to |2J, we evaluate inner bounds on the attainable channel regions 
whose calculation is based on upper bounds on the ML decoding error probability. 

In [22 Section 4], Liu et al. have used special cases of the 1961 Gallager bound to derive a 
simplified algorithm for calculating inner bounds on attainable channel regions. As compared to the 
bounds introduced in [2^, the improvement in the tightness of the bounds presented in Theorems ^ 
andEJis expected to enlarge the corresponding inner bounds on the attainable channel regions. Our 
numerical results referring to inner bounds on attainable channel regions are based on the following 
theorem: 

Theorem 3 (Inner bounds on the attainable channel regions for parallel channels). Let 

us assume that the transmission of a sequence of binary linear block codes (or ensembles) {[C(ra)]} 
takes place over a set of J parallel MBIOS channels. Assume that the bits are randomly assigned to 
these channels, so that every bit is transmitted over a single channel and the a-priori probability for 
transmitting a bit over the j-th channel is ctj (where ^/=i a j = 1 an d ctj > for j G {1, . . . , J}). 

Let {-Ajf^} designate the (average) distance spectrum of the sequence of codes (or ensembles), 
A c \5) designate the asymptotic exponent of the (average) distance spectrum, and 

7; = EvV(y|0;j>(j/|i;j) , je{i,...,J} 

y&y 

designate the Bhattachryya constants of the channels. Assume that the following conditions hold: 

inf E DS2 (5) > 0, V 5q G (0, 1) (86) 

5q<<5<! 
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where, for < S < 1, i? DS2 (<5) is calculated from ()37j) by maximizing w.r.t. A, p (A > and 
< p < 1) and the probability tilting measures {ip(-\ j)}j = i- 

2. The inequality 

AC](X\ I 'JL. \ 

(87) 



limsup = — < — in > a ,-7,- 



is satisfied, where the sum inside the logarithm designates the average Bhattacharrya constant over 
the J parallel channels, and r^ c \5) designates the asymptotic growth rate of the distance spectrum 
as defined in Q14|). 



3. There exists a sequence {D n } of natural numbers tending to infinity with increasing n so that 

limsup^4 C(n)] =0 



n— >oo 

n=l 



4. The normalized exponent of the distance spectrum A c ( n ^ converges uniformly in 5 £ [0, 1] to its 
asymptotic limit. 

Then, the J-tuple vector of parameters characterizing these channels lies within the attainable 
channel region under ML decoding. 

Proof. The reader is referred to Appendix B. □ 

Discussion: We note that conditions 3 and 4 in Theorem|31are similar to the last two conditions 
in |2U1 Theorem 2.3]. Condition 2 above happens to be a natural generalization of the second 
condition in |2()[ Theorem 2.3], thus generalizing the single channel case to a set of parallel channels. 
The distinction between |2U1 Theorem 2.3] which relates to typical-pairs decoding over a single 
channel and the statement in Theorem|2]for ML decoding over a set of independent parallel channels 
lies mainly in the first condition of both theorems. 

A similar result which involves the generalized 1961 Gallager bound for parallel channels can 
be proven in the same way by replacing the first condition with an equivalent relation involving the 
exponent of the 1961 Gallager bound maximized over its parameters, instead of the error exponent 
of the DS2 bound. 

The difference of our results from those presented in |21| stems from the fact that we rely 
here on the generalized DS2 bound and the 1961 Gallager bound with their related optimized 
tilting measures, and not on particular cases of the latter bound. These optimizations which are 
carried over the tilting measures of both bounds provide tighter bounds as compared to the bounds 
introduced in |2H Sections 4 and 5] which follow from the particular choices of the tilting measures 
for the generalized 1961 Gallager bound. 

We later exemplify our inner bounds on the attainable channel regions for ensembles of accumulate- 
based codes whose transmission takes place over parallel AWGN channels. The simplest ensemble 
we consider is the ensemble of uniformly interleaved and non-systematic repeat-accumulate (NSRA) 
codes with q > 3 repetitions. It is shown in Section 5] that the third condition in Theorem 01 is 
satisfied for this ensemble, and more explicitly 

D„ 



h=l V 7 
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where D n = O (ln(n)) (so the sequence {D n } tends to infinity logarithmically with n). Based on 
the calculations of the distance spectrum of this ensemble (see Section 4]), the fourth condition 
in Theorem |31 is also satisfied. We note that for this ensemble, the asymptotic growth rate of the 
distance spectrum satisfies 



Hence, inequality (|87jl in Theorem El (i.e., the second condition in this theorem) is also satisfied for 
this ensemble (since the RHS of (|87|) is always positive). Hence, the fulfillment of all the conditions 
in Theorem 01 for this ensemble requires to check under which conditions the error exponent is 
strictly positive (see the condition in (|86jl). 

As a second example, for the Gallager ensembles of regular (n,j,k) LDPC codes, the second, 
third and fourth conditions are also satisfied for the case where j > 3. Under this assumption, the 
minimum distance even grows linearly with the block length (see 1121 Section 2.2]), so the LHS of 
(jBTj) becomes negative. 

We make use of the fulfillment of the condition in (|87f) for regular NSRA codes and some other 
variants of accumulate-based codes later in Section [7.21 

It is important to note that the low Hamming weight codewords which are addressed by the 
requirement in (|88|) may yield that the error probability under ML decoding does not necessarily 
vanish exponentially with the block length (see, e.g., [22J Theorems 3 and 4] and Section 5], 
where the ML decoding error probability of the considered ensembles of turbo-like codes vanish 
asymptotically like the inverse of a polynomial of the block length). 

Similarly to |2Uj . the condition in (|86|) almost implies the fulfillment of inequality ()87|) . However, 
the former condition in (|86j) may imply the fulfillment of the latter condition in (|87|) with equality. 
This note is clarified in the proof of Theorem (see Appendix B). 

7 Performance Bounds for Turbo-Like Ensembles over Parallel 
Channels: Moderate Block Lengths and the Asymptotic Case 

In this section, we exemplify the performance bounds derived in this paper for various ensembles of 
turbo- like codes whose transmission is assumed to take place over parallel BIAWGN channels. We 
also compare the bounds to those introduced in |2J, showing the superiority of the new bounds 
introduced in Sections [3 and 01 As mentioned before, the superiority of the generalized 1961 
Gallager bound in Section 0] over the LMSF bound from |21j is attributed to the optimizations of 
the related tilting measures over each of the individual channels. 

We especially focus on ensembles of uniformly interleaved repeat-accumulate (RA) codes and 
accumulate-repeat-accumulate (ARA) codes. These codes, originally introduced by Divsalar et 
al. PJEJ) are attractive since they possess low encoding and decoding complexity under iterative 
decoding and remarkable improvement in performance over classical algebraic codes. For indepen- 
dent parallel channels, we study their theoretical performance under ML decoding. The section 
considers both finite-length analysis and asymptotic analysis. In the former case, we present upper 
bounds on the ML decoding error probability, and in the latter case, we consider inner bounds on 
the attainable channel regions of these ensembles and study the gap to the capacity region. In order 
to assess the tightness of the bounds for ensembles of relatively short block lengths, we compare the 
upper bounds under optimal ML decoding with computer simulations under (sub-optimal) itera- 
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tive decoding. In some cases, the upper bounds under ML decoding are more pessimistic than the 
experimental results of the iterative decoder, thus indicating that there is still room for improving 
the tightness of the new bounds. 

The structure of this section is as follows. Section 17.11 exemplifies performance bounds for 
ensembles of short to moderate block length by focusing on a uniformly interleaved ensemble of 
turbo codes, comparing various bounds on the bit error probability under ML decoding and compare 
the results with computer simulation of the Log- MAP iterative decoding. Section 17.21 focuses on 
performance bounds for repeat-accumulate codes and their recent variations which are attractive 
due to their remarkable performance and low encoding and decoding complexity under iterative 
decoding algorithms. This sub-section analyzes the performance under optimal ML decoding, 
assuming the communications takes place over parallel channels. The inner bounds on the attainable 
channel regions whose calculations are based on Theorem |S1 considerably extend the channel region 
which corresponds to the cutoff rate, and outperform previously reported bounds. We conclude 
the discussion in this section with practical considerations related to efficient implementations of 
the generalized DS2 and 1961 Gallager bounds for parallel channels, thus aiming to reduce the 
computational complexity related to the evaluation of these bounds (see Section \7. 3j) . 



7.1 Performance Bounds for Uniformly Interleaved Turbo Codes 

In this sub-section, we exemplify the tightness of the new bounds by referring to an ensemble of 
uniformly interleaved turbo codes, and comparing the upper bounds on the bit error probability 
under ML decoding with computer simulations of an iterative decoder. The bounds for turbo code 
ensembles refer to parallel BIAWGN channels. The reader is referred to |19j which introduces 
coding theorems for turbo code ensembles under ML decoding, assuming that the transmission 
takes place over a single MBIOS channel (i.e., J = 1 in our setting). 

Fig. 0] compares upper bounds on the bit error probability of the ensemble of uniformly in- 
terleaved turbo codes of rate R = g bits per channel use (see Fig. 4(a)). The calculation of the 



average distance spectrum and IOWE of this ensemble is performed by calculating the IOWE of 
the constituent codes which are recursive systematic convolutional codes (to this end, we rely on 
the general approach provided in [23] for the calculation of the IOWE of convolutional codes), 



and finally, the uniform interleaver which is placed between the two constituent codes in Fig. 4(a) 
enables one to calculate the distance spectrum and the IOWE of this ensemble, based on the IOWE 
of the constituent codes (see jH]). The transmission of the codes from this ensemble is assumed to 
take place over two (independent) parallel binary-input AWGN channels where each bit is equally 
likely to be assigned to one of these channels (ax = a.2 = ^), and the value of the energy per bit 

to spectral noise density of the first channel is fixed to (j^j = dB. Since for long enough block 

codes, the union bound is not informative at rates beyond the cutoff rate, one would expect that 
for the considered ensemble of codes (whose block length is roughly 3000 bits), the union bound 

becomes useless for values of (j^j below the value in the RHS of (JSJ) (whose value in this setting 

is 3.69 dB). This limitation of the union bound is indeed reflected from Fig. |4(b)[ thus showing 
how loose is the union bound as compared to computer simulations of the (sub-optimal) iterative 
decoder. The LMSF bound depicted in Fig. |4(b)| uses a partitioning for codes of finite length which 
was obtained via the algorithm described in Section 15.41 for a bit error probability of 10 -4 it is 
about 1 dB tighter than the union bound. The DS2 and the 1961 Gallager bounds with their opti- 
mized tilting measures show a remarkable improvement in their tightness over the union and LMSF 
bounds where for a bit error probability of 10 -4 , the former two bounds exhibit a gain of 0.8 dB 
over the LMSF bound. The DS2 bound and the 1961 Gallager bound are roughly equally tight; 
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the DS2 bound gains about 0.05 dB at a bit error probability of 10 -3 . In spite of a remarkable 
advantage of the improved bounds over the union and LMSF bounds, computer simulations under 
(the sub-optimal) iterative Log-MAP decoding with 10 iterations show a gain of about 0.4 dB, so 
there is still room for further improvement in the tightness of the bounds under ML decoding. 

7.2 Distance Properties and Performance Bounds for Various Ensembles of 
Accumulate-Based Codes 

The IOWEs and the distance spectra of ensembles of irregular repeat-accumulate (IRA) codes and 
accumulate-repeat-accumulate (ARA) codes were derived in I16j . In the continuation of this 
section, we compare inner bounds on the attainable channel regions of accumulate-based codes 
under ML decoding. The comparison refers to three ensembles of rate one-third, as depicted in 
Fig. EJ the first one is the ensemble of uniformly interleaved and non-systematic RA codes where 
the number of repetitions is q = 3, the second and the third ensembles are uniformly interleaved and 
systematic ensembles of RA codes and ARA codes, respectively, where the number of repetitions 
is equal to q = 6 and, as a result of puncturing, every third bit of the non-systematic part is 
transmitted (so the puncturing period is p = 3). For simplicity of notation, we make use of the 
abbreviations NSRA(A r , q), SPRA(A r ,p, q) and SPARA(A r , M,p, q) codes for the encoders shown in 
Figs. [SI (a)— (c) (i.e., the abbreviations 'NS' and 'SP' stand for 'non-systematic' and 'systematic and 
punctured', respectively). In this notation, N is the input block length. 

We rely on the concepts of the analysis introduced in for the calculation of the IOWEs of 
the uniformly interleaved ensembles in Figs. El (a)-(c), as well as the calculation of the asymptotic 
growth rates of their distance spectra. The generalizations of the DS2 and the 1961 Gallager bounds 
for parallel channels are then applied to these ensembles for the asymptotic case where we let the 
block length tend to infinity. 

The average IOWE and the asymptotic growth rate of the distance spectrum for the ensemble 
of uniformly interleaved NSRA codes (see Fig. El (a)) are given in ()16|) and (|17|) . respectively. 

Following the approach of the analysis in [I], we derive the average IOWEs of the uniformly 
interleaved ensembles of SPRA(iV,3,6) and SPARA(iV,M,3,6) codes (see Figs. El (b) and (c)). The 
details of this analysis are provided in Appendix D. In the following, we present the final results 
related to the finite- length and asymptotic distance properties (where we let N tend to infinity); 
these results serve later for the calculation of attainable channel regions under ML decoding. 

We consider here the case where the repetition code repeats each bit six times, and the parity 
bits are punctured so that every third bit is transmitted. The IOWE of the ensemble of uniformly 
interleaved SPRA(iV, 3,6) codes in Fig. El (b) is given by (see Appendix ID. 1|) 



where 5 n ^ m designates the discrete delta function (i.e., it is equal to 1 if n = m, and is equal to 
zero otherwise). The IOWE of the ensemble of uniformly interleaved SPARA(iV, M, 3, 6) codes in 




(89) 
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Fig- El (c) is given by (see Appendix ID 
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The asymptotic growth rates of the distance spectra of these two ensembles are obtained by the 
calculation of the limit 

1 N a 
r^(<5) = lim — V A wd , 5 = -- (0 < 5 < 1) 

w=0 

where it is taken into account that the common rate of these ensembles is equal to one-third bits 
per channel use. 

Let H(x) = — xln(x) — (1 — x)ln(l — x) be the binary entropy function to the natural base, 
then it is well known that 

lim - In f " J = H{B) , < f3 < 1. (91) 

n-+oo n \pn J 

For the derivation of the asymptotic growth rate of the average distance spectrum for the 
ensemble of uniformly interleaved SPRA (iV, 3, 6) codes where N — ► oo, we rely on the IOWE given 
in Q89|) . and introduce the parameters 

_A h _A j _A 3 

^'W' Pl ~3iV' P2 ~3iV' 
The asymptotic growth rate of the distance spectrum of this ensemble is given by (see Appendix lD.2|) 
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where the three-parameter maximization is performed over the finite domain which is characterized 
by the following inequalities: 

< 77 < |, 0<p 2 <^, 2p 2 + r]<65, p 2 + 2ry < 35 

max(0,p 2 + ?? - ^) < Pi < min(/32,r/), rj - p 2 + 35 < 2. (93) 
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Finally, the derivation of the asymptotic growth rate of the average distance spectrum for the 
ensemble of uniformly interleaved SPARA (N, M, 3, 6) codes, where iV — ► oo and the ratio is 
fixed, relies on the IOWE given in ()9U|) . To this end, we introduce the three additional parameters 



a M A m A w-m 

a = 3iv' £l = 3iv' £2 = ^r- (94) 

As mentioned above, the value of a is fixed, and also < a < | (since M < N). After straight- 
forward and tedious algebra which is conceptually similar to the calculations in Appendix ID. 21 one 
obtains the following expression for the asymptotic growth rate of the average distance spectrum 
of the considered ensemble of uniformly interleaved SPARA codes: 

fx\ \ uf £l \ , ( l 2 P2 + V \ e 2 
r(b) = max <aH[ — + - - a h E\ ] H 1 
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3 ') \ \-n J ' \s ^ "V y 2 (l-s + s 1 + s 2 ) J 

+(5 - e 1 - e 2 )H (^-^——^j + (r, + p 2 - 2 Pl ) lnsj (95) 

where the five-parameter maximization is performed over the finite domain which is characterized 
by the following inequalities: 

< V < \, < p 2 < |, < ei < a, 
< st + e 2 < min^, 
max(o, p 2 + rj - - ) < p± < min(p 2 , 77), 



3 

< 77 < min(- - 25 + 2(ei + e 2 ), 25 - 2( £l + e 2 ) 

, . /2 2/j 2 +r/ 2p 2 + rj , 
e 2 < mm I - - 2a h 2ei, 2e x ) . (96) 



In the asymptotic case where we let the block length tend to infinity, inner bounds on the 
attainable channel regions for the considered ensembles of accumulate-based codes are calculated 
in this section by Theorem OJ 

In Fig. we compare inner bounds on the attainable channel boundaries as calculated by the 
union, LMSF, 1961 Gallager and DS2 bounds. These plot refers to the ensemble of NSRA(A r , 3) 
codes of rate | bits per channel use (see Fig. 0(a)) where we let N tend to infinity. The asymptotic 
growth rate of the distance spectrum of this ensemble is calculated by (|17[) with q = 3. The 
remarkable superiority of the generalized DS2 and 1961 Gallager bounds over the union and LMSF 
bounds is exemplified for this ensemble of turbo-like codes; actually, the 1961 Gallager bound 
appears to be slightly tighter than the generalized DS2 bound at the extremities of the boundary of 
the attainable channel region. This phenomenon was also observed for various turbo-like ensembles, 
as well as for ensembles of fully random block codes. In the continuation of this section, we therefore 
compare inner bounds on the attainable channel regions for various ensembles of turbo-like codes 
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where the boundaries of these regions are determined via the generalization of the 1961 Gallager 
bound (see Fig. EJ). 

Fig. compares the asymptotic growth rate of the distance spectra of several ensembles of 
uniformly interleaved and accumulate-based codes where the ensembles are depicted in Fig. |21 The 
improved performance of the ensembles of SPARA codes under ML decoding is demonstrated by 
the Gallager bounding technique in Fig. El This improvement is attributed to the distance spectral 
thinning effect [25] which is exemplified in Fig. for the ensembles of NSRA, SPRA and SPARA 
codes of the same code rate (| bits per channel use). The same phenomenon of distance spectral 
thinning also exists by reducing the value of a for the ensembles of SPARA codes (see Fig. [7J 
comparing the two plots for a = j and a = j^); this in turn yields an improved inner bound on 
the attainable channel regions, as observed in Fig. El It is shown in this figure that for the SPARA 
ensemble with the parameters p = 3,q = 6 and a = fjr, the gap between the inner bound on the 
attainable channel region under ML decoding and the capacity limit is less than 0.05 dB. Note that 
for the examined ensembles of NSRA and SPRA codes of the same code rate, the corresponding 
gaps between the inner bounds on the attainable channel regions and the channel capacity are 

2.2 dB and 0.5 dB, respectively (see Fig. EJ). 

7.3 Considerations on the Computational Complexity of the Generalized DS2 
and 1961 Gallager Bounds 

The brute-force calculation of the generalized DS2 bound for linear codes (or ensembles) of finite 
length is in general computationally heavy. For every constant weight subcode, it requires a nu- 
merical optimization over the two parameters A > and < p < 1; for each subcode of constant 
Hamming weight and for each choice of values for A and p, one needs to solve numerically the 
explicit equations for k and (5j (see Eqs. (|43j) and 1)441) ) which are related to the J optimized tilting 
measures. Moreover, for each subcode and a pair of values for A and p, the evaluation of the 
generalized DS2 bound requires numerical integrations (or summations, in case the channel out- 
puts are discrete). Performing these tedious and time-consuming optimizations for every constant 
weight subcode would make the improved bounds less attractive in terms of their practical use for 
performance evaluation of linear codes and ensembles. 

In the following, we suggest an approach which significantly reduces the complexity related to 
the computation of the generalized DS2 bound, and enhances the applicability of the bound using 
standard computational facilities. First, the code is partitioned into constant Hamming weight 
subcodes, and the exact union bound (see Eq. (jC.5)) ) is calculated for every subcode (note that the 
number of subcodes does not exceed the block length of the code). This task is rather easy, given the 
(average) distance spectrum {A^} or the weighted IOWE {A' h } of the code (or ensemble) which are 
calculated in advance (see (|2*H|) and (|2*4"jl ). In order to reduce the computational complexity, we do 
not calculate the generalized DS2 bounds for those constant-Hamming weight subcodes for which 
the values of the union bounds are below a certain threshold (e.g., we may choose a threshold of 
10~ 10 for bit error probability or 10~ 6 for block error probability; these thresholds should be tailored 
for the application under consideration). Next, for those constant Hamming weight subcodes for 
which the union bound exceeds the above threshold, the generalized DS2 bound is evaluated. For 
these subcodes, we wish to reduce the infinite interval A > to a finite interval; this is performed by 
using the transformation A' = -^-j- so that the two-parameter optimization is reduced to a numerical 
optimization over the unit square (A',p) G [0, l] 2 . In this respect, it was observed that the optimal 
values of A' and p vary rather slowly for consecutive values of the constant Hamming weight h, so 
the search interval associated with the optimization process may be reduced once again with no 
penalty in the tightness of the bound. In other words, we search for optimal values of A' and p 
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only within a neighborhood of the optimal A' and p found for the previous subcode. We proceed 
in this manner until all the relevant subcodes are considered. As an example, we note that for the 
ensemble of turbo codes depicted in Fig. |4(b)| about 80% of the computational time was saved 
without affecting the numerical results; in this respect, the threshold for the bit error probability 
analysis was chosen to be where n designates the block length of the code. The reduction in 
the computational complexity becomes however more pronounced for higher SNR values, as the 
number of subcodes for which the union bound replaces the computation of the generalized DS2 
bound increases. 

An analogous consideration applies to the generalized version of the 1961 Gallager bound for 
parallel channels with its related optimized tilting measures. 

Referring to the calculation of attainable channel regions, a search over the region of channel 
parameters is required. As an example, consider a set of parallel AWGN channels characterized 
by the J-tuple of SNRs (fx, ... , vj). In order to find the attainable channel boundary, we fix the 
values of v\, . . . , vj—\ and perform a linear search over uj using any appropriate method (e.g., 
the bisection method) in order to find the smallest value of v*j for which the lower bound on the 
error exponent (as obtained by an upper bound on the ML decoding error probability) vanishes. If 
(yi, . . . , vj-i, 0) is not an attainable point while . . . , vj-x, oo) is attainable, then the resulting 
value v*j is such that the point (i>\, . . . , Vj) is on the boundary of the attainable region. The overall 
complexity of this approach is, of course, polynomial in J. We apply this approach in this section 
for the calculation of inner bounds on the attainable channel regions under ML decoding, referring 
to the generalizations of the DS2 and 1961 Gallager bounds in Sections |3] and respectively. 



8 Summary 



This paper is focused on the performance analysis of binary linear block codes (or ensembles) whose 
transmission takes place over independent and memoryless parallel channels. New bounds on the 
maximum-likelihood (ML) decoding error probability are derived. These bounds are applied to 
various ensembles of turbo-like codes, focusing especially on repeat-accumulate codes and their 
recent variations which possess low encoding and decoding complexity and exhibit remarkable 
performance under iterative decoding (see, e.g., ^ [3 [26 ]). The framework of the second 
version of the Duman and Salehi (DS2) bounds is generalized to the case of parallel channels, along 
with the derivation of their optimized tilting measures. The generalization of the 1961 Gallager 
bound for parallel channels, introduced by Liu at al. [2j|, is reviewed and the optimized tilting 
measures which are related to this bound are calculated via calculus of variations (as opposed to 
the use of simple and sub-optimal tilting measures in ) . The connection between the generalized 
DS2 and the 1961 Gallager bounds, which was originally addressed by Divsalar [B] and by Sason 
and Shamai |29l l32 for a single channel, is explored for an arbitrary number of independent parallel 
channels. In this respect, it is shown that the generalization of the DS2 bound for independent 
parallel channels is not necessarily tighter than the generalization of the 1961 Gallager bound |21j . 
as opposed to the case where the communication takes place over a single channel; this difference 
is attributed to the additional Jensen's inequality in the transition from l)28|) to (|29|) (see p. 113}) 
which is circumvented in the generalization of the 1961 Gallager bound for parallel channels (see 
(|57l) in p. 1201 where the upper bound is linear in the split weight enumerator, thus there is no need 
for an additional use of the Jensen's inequality due to the random mapper). 

In the asymptotic case where we let the block length tend to infinity, the new bounds are 
used to obtain improved inner bounds on the attainable channel regions under ML decoding. 
The tightness of the new bounds for independent parallel channels is exemplified for structured 
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ensembles of turbo-like codes. In this respect, the inner bounds on the attainable channel regions 
which are computed by the generalized DS2 bound are slightly looser than those computed by the 
generalized 1961 Gallager bound. On the other hand, for turbo-like ensembles of moderate block 
lengths, the generalized DS2 bound for parallel channels appears to be slightly tighter than the 
generalized 1961 Gallager bound (see, e.g., Fig. |4(b)| in p. I55[). This enhances the conclusion that 
there is no sharp advantage in the tightness of one bound over the other. 

Following the approach in pQ, we analyze the distance spectra and their asymptotic growth 
rates for various ensembles of systematic and punctured accumulate-based codes (see Fig. [SJ) . This 
distance spectral analysis serves to assess the performance of these codes under ML decoding where 
we rely on the bounding techniques developed in this paper and in [2^ f° r parallel channels. The 
improved performance of the ensembles of systematic and punctured accumulate-repeat-accumulate 
(SPARA) codes under ML decoding is demonstrated by the Gallager bounding technique in Fig. El 
This improvement is attributed to the distance spectral thinning effect which is exemplified 
in Fig. by comparing the asymptotic growth rates of the distance spectra for the ensembles in 
Fig.EKa)-(c). 

The generalization of the DS2 bound for parallel channels enables to re-derive specific bounds 
which were originally derived by Liu et al. |2j as special cases of the 1961 Gallager bound. The 
improved bounds with their optimized tilting measures show, regardless of the block length of the 
codes, an improvement over the union bound and other previously reported bounds for independent 
parallel channels; this improvement is especially pronounced for moderate to large block lengths. 
However, in some cases, the new bounds under ML decoding happen to be a bit pessimistic as 
compared to computer simulations of sub-optimal iterative decoding (see, e.g., Fig. |4(b)| ), thus 
indicating that there is still room for further improvement. 
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Appendices 



Appendix A 



A.l On the Sub-optimality of Even Tilting Measures in the Gallager Bound 



In the following, we derive the functions /(•; j) resulting from the optimal DS2 tilting measures in 
(j42|) and demonstrate that they are not even functions. From (|32|) . we get the expression 

ip{y;j) = -=c ■ g(y,j)p(y\0;j) 

2_^g(y ;j)p(y |0;i) 

y' 

for the single-letter connection between the normalized and un-normalized DS2 tilting measures; 
changing the subject gives 

Substituting (|42|) in (|A.1|) we obtain the optimal form of the un-normalized tilting measure as 



j?(y|0;j 

Next, we substitute (|65[) in the LHS of (|A,2|) and manipulate the expression to get 

(p 
i+ KrkI)T ■ (A,3) 

Clearly, this expression does not constitute an even function. 



A.2 Some Technical Details Related to the Derivation of the Bound in (jHTjl 



In the following, we use the relation between the 1961 Gallager bound and the DS2 bound in order 
to get a bound similar in form to the DS2 bound l)31j) which is tighter than ()66|) . We begin with 
the bounds in (|52|) and (|53|) and average their sum over all possible channel assignments, yielding 
that 
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where G(- and V(-, ■ ;j) are denned in for j = 1, . . . , J. Following the same steps as in 
ODMEJ, we get 
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Optimizing this bound over the parameter d (where d € H.) gives 
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Substituting (jHSJ) in (jSU) yields 

G(s;j) = ^2p{y\0;j)g(y,j) 
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. ^ l- - 
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Finally, substituting (jA.7|) in (|A.6|) gives 
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We note that this bound is tighter than (|66|) . because we have not invoked Jensen's inequality in 
the process of averaging over all channel assignments in (|A,4|) . 



A. 3 Technical Details for Calculus of Variations on (fTTj) 

The bound on the decoding error probability for constant Hamming weight codes is given by 
substituting (|70|) into (|58[) , Disregarding the multiplicative term 2 h ( p \ we minimize the expression 
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Employing calculus of variations, we substitute in (jA.8|) the following tilting measure 

f(y;j) = fa(y;j) + £v(y;j) 

where is an arbitrary function. Next, we impose the condition that ^r| e _ = for all ??(•; j). 

The derivative is given by 
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Defining the constants 
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and requiring that the integrand in (IA,9|1 be equal to zero, we get the equivalent condition 
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Defining K% = ^r 1 , K2 = ^r 1 , and dividing both sides by fo(y,j) r 1 implies the condition in ((72)) . 



Appendix B 



Proof of Theorem 02 



The concept of the proof of this theorem is similar to the proofs introduced in |201 pp. 40-42] 
for the single channel case, and the proofs of |21l Theorems 2-4] for the scenario of independent 
parallel channels. The difference in this proof from those mentioned above is the starting point 
which relies on the generalization of the DS2 bound (see Theorem^ 11 SectionEJ). Like these proofs, 
the essence of the proof is showing that apart from the requirement in (|86|) . the lower bound on 
the error exponent in (|37j). which follows here from the generalized DS2 bound, is positive and 
behaves linearly in 5 for small enough values of 5. To this end, let us first verify from (|43|) that the 
partial derivative of the optimized parameter k w.r.t. 5 is strictly positive at 5 = 0. This follows 
by rewriting the implicit equation for k in Q43JI as 



k 



5 h(k) 
l-Sg(k) 



where h and g stand for the numerator and denominator of the term which multiplies in the 
RHS of (@HJ). Note that (@HJ) also implies that k = at 5 = 0, and hence at 5 = 0, (jSJ) gives that 
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f3j = 1 for all j G {1, . . . , J}. Differentiating both sides of (|43[) w.r.t. 5 and setting <5 = gives 



dk 
06 



5=0 



(B.l) 







= < 




5=0 





From (|87|). it follows that limsup^g rt c l(5) < 0, so in the limit where 5 — > 0, the Bhattacharyya 
union bound becomes tight for the conditional ML decoding error probability (given that the all-zero 
codeword is transmitted) w.r.t. subcodes of constant Hamming weight with normalized Hamming 
weight S. Hence, p — ► 1 and A — > | become optimal in the limit where 5^0. The substitution of 
A = \ in the RHS of (lB~Tl) gives 



3fc 
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where 7j is the Bhattacharyya parameter of the j-th channel. 

For values of 5 close enough to zero, let us analyze the behavior of each of the three terms in 
the RHS of JH3>. Let 

r =limsup— ^ (B.2) 

then the first term in the RHS of (|3"7j) behaves like — r^pd for values of 5 close enough to zero, i.e., 

- pr t c ] (5) = -r p5 + 0(5 2 ). (B.3) 

As for the second term in the RHS of (|37j) . since k tends to zero and (3j —* 1 for all j as we let 
5 — > 0, then the optimized tilting measures vp(y;j) in ()42|) tend to the conditional pd/s p(y|0;j), 
respectively, for all j G {1, . . . , J}. Since p — ► 1 and A — > ^ are optimal as we let <5 — > 0, then in 
the limit where 5 is close enough to zero, we get 



-pSln I 2_^aj 2_^ip(y;j) >>p{y\0;j) " p(y\l;j) 
\i=i y^y 

-- -pdin ( J2 a iJ2 Vp(y\0;j)p(y\hj) ) +o(S) 

-- -p5hx I Yl a ^A +°( 6 )- 



(B.4) 



As we let 5 tend to zero, we will show that the third term in the RHS of ()37j) converges to 
zero quadratically in 5. To this end, we rely on the expression for the optimized tilting measure 
in 1)42(1 and the linear behavior of k near 5 (from (|B.1|) and since k = for 5 = 0, it follows that 
k = rj5 + 0(5 2 )). We obtain from (|42|) that for values of 5 close enough to zero, the optimized 
tilting measure of the generalized DS2 bound behaves like 



i>(y,j)=P 3 p(y\0;j) ( i + W^^y I +°^ 2 ) ( B - 5 ) 
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where j3j is the normalization factor which is calculated by (|44|l and is given by @j = ■ Hence, 

for small values of 5 which are close enough to zero, the third term in the RHS of ()37j) behaves like 



J 



-p(i -S)]n\J2 oij ^2 i) 1 p p(y\0; j) » 

i i=i yey 



® -p(l - 6) In ( £ a, £(1 + J) 1 + | + O(^) 



= - 8) In (l + V (- p - l) 7 ^) (l - V (- p - l) 7j 5) j + 0(5 2 ) 

= - *) ^ ( £ a, (l - ^ - l) %|5 2 ) ) + 0(5 2 ) 



V J=1 

2 



-p(l - 5) In 1 - 7/ 



( = } P(l - 5W (- - l) 2 £ a i7 | 5 2 + 0(^ 2 ) = 0(5 2 ) (B.6) 

where (a) follows from ()B.5|) . (b) follows from the equality 

(1 + e£) c = 1 + eC<5 + 0(<5 2 ) , Ve,CeM 

(c) follows from the definition of the Bhattacharyya constant jj for the j-th channel and since 
^y&y P(y\®'ij) = 1> anc ^ (^) follows from the equality ln(l + cc) = x + 0(x 2 ). Hence, as we let 
5^0, the expression in ()B.6|) converges to zero quadratically in 5. 



From ()B.3|) - ()B.6|) . one obtains that for values of 5 which are close enough to zero, the error 
exponent in (|37jl decays linearly with 5 if and only if 

-r - In ^E^'j > 

which coincides with the condition in (|87jl (vq is defined in (|B.2|0 . This is indeed the second 
requirement in Theorem By combining it with the requirement in (|86|) . we obtain that the ML 
decoding error probability vanishes as we let n — > 00, as long as we exclude the codewords whose 
Hamming weights behave sub-linearly with the block length n. By showing that the error exponent 
of the generalized DS2 bound grows linearly with 5 as we let 5 — > and due to the requirement 
in (|88|) . it follows that asymptotically in the limit where the block length tends to infinity, these 
codewords contribute a vanishing effect to the ML decoding error probability (similarly to the 
proofs of 20, Theorem 2.3] and |211 Theorems 2-4]). 
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Appendix C 



Exact Union Bound for Parallel Gaussian Channels 



In this appendix, we derive the union bound on the ML decoding error probability of binary 
linear block codes transmitted over parallel Gaussian channels. This form of the union bound will 
be used in conjunction with other bounds (e.g., 1961 Gallager or the DS2 bounds) for constant 
Hamming weight subcodes in order to tighten the resulting bound. 

We start the derivation by expressing the pairwise error probability given that the all-zero 
codeword is transmitted 



h 2 ,...,h.i 



Q 



\ i=i 



(C.l) 



where x hlMj ^ hj 
channels, and v« 



is a codeword possessing split Hamming weights h\,...,hj in the J parallel 



A l Ms. 

N 



designates the energy per symbol to spectral noise density for the j 



th 



AWGN channel (J = 1, 2, . . . , J). The union bound on the block error probability gives 

Ahi,...,hjQ 



p = £ E E 



h=l hi>0,...,hj>0 
h\+...+hj=h 



(C.2) 



where this bound is expressed in terms of the split weight enumerator of the code. Averaging (|C.2|) 
over all possible channel assignments and codes from the ensemble gives (see 1)30 Jl ) 



?e < £ « 

rij>0 
n 1 +...+n J =n 

- E 

n 3 >0 
ni+...+nj=n 



E E Ah PH\N(h\n) PN_(n) Q 

0<hj<n,j 
Ej h j= h 



E E ^ " 



hi, h,2, ■ ■ ■ , hj 



n — h 

n x - hi,n 2 - h 2 , . . . ,nj - hj 



a. 



otjQ 



\ 3=1 



(C.3) 



where atj designates the a-priori probability for the transmission of symbols over the j th channel, 
assuming the assignments of these symbols to the J parallel channels are independent and random. 



In order to simplify the final result, we rely on Craig's identity for the Q-function, i.e., 



Q(x) 



7T 



e J^n^dO , x > 0. 



(C.4) 
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Plugging (|C4|) into (|C3j) and interchanging the order of integration and summation gives 



Po < 



1 /"a 



7T 



E 



ni+...+nj=n 



E E ^ 



hi,h,2, ■ ■ ■ ,hj 



E ^ =?i 



n — h 

m - h\,ri2 - h 2 , ■ ■ ■ ,nj - hj 



J 



Til Hi 1 | . i) 

a 1 . . . a j | | e ™ 2 



n 

3=1 



\ d6 



(a) 1 
7T 



2L n 
2 



E^ E 

h 3 >C 
E hj— h 

E 

fc.,>o 

Ej kj=n-h 



h 

hi,fi2, ■ ■ ■ ,h u 



n[ 



(b) 1 

7T 



2L n 

2 



E^ 



ft 



ft=i 



E 



a^e sin 2 s 



de 



(C.5) 



where (a) follows by substituting kj = rij — hj for j = 1, 2, . . . , J, and (b) follows since the sequence 
0= 



{ay}/=i is a probability distribution, which gives the equality 



E 



n — h 
ki,k 2 ,...,kj 



Ej kj=n-h 



) JI(^} = (E«ij =i- 



Eq. (j0.5D provides the exact (Q-form) version of the union bound on the block error probability 
for independent parallel AWGN channels. 



Appendix D 



Distance Spectra Analysis of Systematic Accumulate-Based Codes with 

Puncturing 

The following analysis is focused on the distance spectra of uniformly interleaved and systematic 
ensembles of repeat-accumulate (RA) codes and accumulate-repeat-accumulate (ARA) codes with 
puncturing (see Figs. 03 (b) and (c) in p. l56|). As mentioned in Section E3 these two ensembles 
are abbreviated by SPRA and SPARA codes, respectively (where 'SP' stands for 'systematic and 
punctured'). We derive here the input-output weight enumerator (IOWEs) of these ensembles 
and also calculate the asymptotic growth rates of their distance spectra. The analysis follows the 
approach introduced in pQ, and it is written in a self-contained manner. 

The component codes constructing SPRA and SPARA codes are an accumulate code (i.e., a 
rate-1 differential encoder), a repetition code and a single parity-check (SPC) code. Since we 
consider ensembles of uniformly interleaved codes, their IOWEs depend on the IOWE of the above 
component codes [Hill]- As a preparatory step, we introduce the IOWEs of these components. 

1. The IOWE of a repetition (REP) code is given by 

C <5) = (*) ^ 
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where k designates the input block length, and 8 n , m is the discrete delta function. 

2. The IOWE of an accumulate (ACC) code is given by 

where n is the block length (since this code is of rate 1, the input and output block lengths are the 
same). The IOWE in (|D.2|) can be easily obtained combinatorially; to this end, we rely on the fact 
that for the accumulate code, every single '1' at the input sequence flips the value at the output 
from this point (until the occurrence of the next '1' at the input sequence). 

3. The IOWE function of a non-systematic single parity-check code which provides the parity bit of 
each set of p consecutive bits, call it SPC(p), is given by (see Eq. (8)]) 

rap n 

A(W,D) = ^^i^W 

w=0 d=0 

Even((l + IW) p ) +0&&({l + Wf)D (D.3) 
(1 + W)p + (1 - W)p 



where 



Even((l + VF) P ) 



2 



0M ( { l + WY) = {l + W) '- 2 {l - W) ' (DA) 

are two polynomials which include the terms with the even and odd powers of W, respectively. 

To verify (|D.3|) . note that a parity-bit of this code is equal to 1 if and only if the number of ones 
in the corresponding set of p bits is odd; also, the number of check nodes in the considered code is 
equal to the block length of the code (n). 

The case where the output bits of an accumulate code are punctured with a puncturing period 
p is equivalent to an SPC(p) code followed by an accumulate code (see Fig. |H]which was originally 
shown in ^ Fig. 2]). Hence, for the uniformly interleaved ensembles of SPRA and SPARA codes 
with a puncturing period of p = 3 (see Figs. El (b) and (c)), we are interested in the IOWE of the 
SPC(3) code. For the case where p = 3, QD.4|) gives 

Even((l + WO 3 ) = 1 + 3VF 2 , Odd((l + Wf) = 3W + W 3 

and (|D.3|) gives straightforwardly the following IOWE of the SPC(3) code [IJ Eq. (15)]: 

<?<M E E I w cm) 



In the following, we consider two uniformly interleaved ensembles of SPRA and SPARA codes 
with q = 6 repetitions and puncturing of period p = 3, as shown in Fig. |S] (b) and (c). We rely 
here on the equivalence shown in Fig. EH related to the inner accumulate code with puncturing. In 
this respect, since the input bits to the SPC (appearing in the right plot in Fig. |I5J) are permuted 
by the uniform interleaver which is placed after the repetition code (see Figs. El (b) and (c)), then 
the average IOWEs of these two ensembles remain unaffected by placing an additional uniform 
interleaver between the SPC and the inner accumulate codes. Similarly, placing another uniform 
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interleaver between the precoder in Fig. [51(c) (i.e., the code which accumulates the first N—M bits) 
and the repetition code, does not affect the average IOWE of the overall ensemble in Fig. El (c). 

As mentioned above, the equivalence in Fig.|2lyields that without loss of generality, an additional 
uniform interleaver of length N' = — = 2N bits can be placed between the SPC(3) code and the 
accumulate code without affecting the calculation. By doing so, the average IOWE of the serially 
concatenated and uniformly interleaved ensemble whose constituent codes are the SPC(3) and the 
accumulate codes, call it ACC(3), is given by (see 0j) 

2N 4SPC(3) ,ACC 
h=0 

The substitution of (|D.2J> and ()D.5|) into ()D.6j) gives 

2N 2N min(j,/i) 



(?) 



■C 0(3, = £E E • 

h=0 j=0 i=max(0,j-2V+/i) 



h\ (2N - h\ (2N -d\ ( d - 1 

i 



j-i J\ lij J vrn-i, 

Z h+j - M 5 Wj2j+h \. (D.7) 



Note that (|D.7|) is similar to ^ Eq. (19)], except that N in the latter equation is replaced by 2N 
in (jD.7j) . This follows since ^ (i.e., the ratio between the number of repetitions and the puncturing 
period) is equal here to 2, instead of 1 as was the case in p. Equation (|D.7|) will be used for the 
finite-length analysis of the distance spectra for the ensembles considered in the continuation to 
this appendix. 



D.l Finite-Length Analysis of the Distance Spectra for Systematic Ensembles 
of RA and ARA Codes with Puncturing 

Uniformly Interleaved SPRA (N, 3, 6) codes: Let us consider the ensemble depicted in Fig. (b) 
where q = 6 and p = 3. Since there is a uniform interleaver of length N" = qN between the 
repetition code and the equivalent ACC(3) code, the average IOWE of this serially concatenated 
and uniformly interleaved systematic ensemble is given by 

67V ,REP(6) , ACC(3) 
,SPRA(Af,3,6) ^Vl ^l,d-w 

A w,d ~ 2.^1 (6N\ 

1=0 \ I ) 

/ATn^ACC(3) 

- J6N\ i D ' 8 ) 

\6w) 

where the last equality is due to the equality in (|D,1|) . Substituting ()D,7|) in the RHS of ()D.8|) gives 
the average IOWE of the considered ensemble, and this result coincides with (|89|) . 

Uniformly Interleaved SPARA (N, M, 3,6) codes: By comparing Figs. (b) and (c), we see that 
a precoder is placed in the second figure. Referring to the ensemble of SPARA codes which is 
shown in Fig. (c), the precoder is a binary linear block code whose first N — M input bits are 
accumulated and the other M input bits not changed; these bits are encoded by the repetition 
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code. The IOWE of the systematic precoder, call it Pre(A, M), is given by 



,Pre(7V,M) _ f M\ Acc 

A w,d - 2^[ m ) A 



m=0 
M 

£ 

m=0 



w—mA—m 



M\ f N — M — d + m\ f d — m — l 



(D.9) 



where the last equality relies on (|D,2|) . As mentioned before, for the uniformly interleaved SPARA 
ensemble depicted in Fig. El (c), an additional uniform interleaver between the precoder and the 
following stages of its encoder does not affect the average IOWE; this ensemble can be therefore 
viewed as a serial concatenation with a uniform interleaver of length N which is placed between the 
precoder and the repetition code in Fig. [51(c) (in addition to the uniform interleaver which is placed 
after the repetition code). Moreover, referring to the systematic ensemble whose components are 
REP (6) and ACC (3), the input bits (which are the bits provided by the precoder to the second 
stage in Fig. |S] (c)) are not transmitted to the channel. In light of these two observations, the 
average IOWE of the uniformly interleaved ensemble of SPARA codes shown in Fig. El (c) is given 
by 

AT A Pre(N,M) a SPRA(iV,3,6) 

,SPARA(7V,M,3,6) _ A w,l A l,d-w+l m -. „,. 

A vi,d ~ 2^ ?N\ • [D.LV) 

1=0 \l) 

By substituting (|D.8|) (i.e., the equality in (|89jll and ()D.9|) into (jD.lOj) . one obtains the expression 
in PI for the average IOWE of the SPARA(A, M, 3, 6) codes. 



D.2 Asymptotic Analysis of the Distance Spectra 

This subsection considers the calculation of the asymptotic growth rates of the distance spectra 
for the two ensembles in Figs. El (b) and (c). The calculation of the asymptotic growth rate of the 
distance spectrum of a sequence of codes (or ensembles) is performed via (|14jl. In the following, we 
exemplify the derivation of ()92|) from the average IOWE in ()89|) . The derivation of (|95j) from (|90j) 
is conceptually similar, but is more tedious algebraically. Since we focus here on ensembles of rate 
one-third and the block length of the input bits is N (see Fig. , the asymptotic growth rate of 
their distance spectra is obtained by normalizing the logarithm of the average distance spectra of 
the considered ensemble by n = 3N and letting N tend to infinity. Referring to the average IOWE 
of the uniformly interleaved ensemble of SPRA(A, 3,6) codes, as given in ()H9|). we introduce the 
normalized parameters 

c A d A h A ^ A 3 ; ni1 i 

3N 3N 3N 3N v ; 

The normalization by 3N yields that the new parameters satisfy 

< (5 < 1, < 77 < |, 0<pz<|. (D.12) 

From the partial sum w.r.t. the index i in the RHS of (|89jl. dividing the terms in the inequality 

max(0, j — 2N + h) < i < min(j, h) 

by 3A gives 

max^0,/J 2 + V ~ < Pi < min(/9 2 ,??). (D.13) 
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Since the codes are systematic and the block length of the input bits is N, we get that the terms 
which contribute to the IOWE in the RHS of ((HSJ) satisfy 



w < mm(d, N), 6w = 2j + h 
and, from (|D.11|) . multiplying ()D.14|) by ^ gives 

< mm o, — . 

From the binomial coefficients which appear in the RHS of (|89|). it is required that 

2N - d + w > 



(D.14) 



(D.15) 





h 


, d — w > 


"ft" 


> 








2 




2 



so dividing both sides of these inequalities by 3N, and letting N tend to infinity gives 

7] - p 2 + 3d < 2, P2 + 2t 1 < 3d. 
Combining (|D.12|) - (|D,16|) gives the domain for the three parameters ry, p\ and p2 in H93|) . 
A marginalization of the IOWE enables one to obtain the distance spectrum 



(D.16) 



A? 



w,d 



w=0 



where the IOWE {^4^^} is given by (|89j) . Note that unless 

w 2j + h 2p 2 + fj 



N 



6iV 



(D.17) 



(D.18) 



the IOWE A 

w,d i n (189(1 vanishes, and therefore it does not affect the sum in the RHS of (|D.17|) . 
In the limit where N — > oo, the asymptotic growth rate of the average distance spectrum for the 
uniformly interleaved ensemble of SPRA(iV, 3, 6) codes (see Fig. |21 (b) in p. I56|) is obtained from 
(gUJ), and ()D.17|) . Hence, we get 



r(6) 



lim 

N^oo 3N 



lim max< 



1 N 



w=0 



\ 1 








NH 




[3A^ 







6w 
6N 



-(2N - d + w)H 
-(h + j - 2i)ln3 



2(2N -d + w) 



+ (d-w-l)H 



2N - h 



d — w — 1 



By multiplying the three parameters which are involved in the maximization by using (jD. lljl 
and ()D.18|) and taking the limit where N tends to infinity, one readily obtains the result in (|92|) . 
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(a) Ensemble of uniformly interleaved turbo codes. 
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(b) Performance Bounds under ML decoding versus simulation 
results of iterative Log-MAP decoding. 



Figure 4: (a) The encoder of an ensemble of uniformly interleaved turbo codes whose interleaver is 
of length 1000, and there is no puncturing of parity bits, (b) Performance bounds for the bit error 
probability under ML decoding versus computer simulation results of iterative Log-MAP decoding 
(with 10 iterations). The transmission of this ensemble takes place over two (independent) parallel 
binary-input AWGN channels. Each bit is equally likely to be assigned to one of these channels, 
and the energy per bit to spectral noise density of the first channel is set to ffj-M = dB. The 

compared upper bounds on the bit error probability are the generalizations of the DS2 and 1961 
Gallager bounds, the LMSF bound from 21J, and the union bound (based on (|C.5|l ). 
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(a) Non-systematic RA codes - NSRA (TV, q) 
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(b) Systematic RA codes with puncturing - SPRA (TV, p, q) 
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(c) Systematic ARA codes with puncturing - SPARA (TV, M, p, q) 



Figure 5: Systematic and Non-systematic RA and ARA codes. The interleavers of these ensembles 
are assumed to be chosen uniformly at random, and are of length qN where N designates the 
length of the input block (information bits) and q is the number of repetitions. The rates of all the 
ensembles is set to | bits per channel use, so we set q = 3 for figure (a), and q = 6 and p = 3 for 
figures (b) and (c). 
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Figure 6: Attainable channel regions for the rate one-third uniformly interleaved ensemble of 
NSRA(iV, 3) codes (see Fig. [S] (a)) in the asymptotic case where we let N tend to infinity. The 
communication takes place over J = 2 parallel binary-input AWGN channels, and the bits are 
equally likely to be assigned over one of these channels {ot\ = «2 = \)- The achievable channel 
region refers to optimal ML decoding. The boundaries of the union and LMSF bounds refer to the 
discussion in [2Tj, while the boundaries referring to the DS2 bound and the 1961 Gallager bound 
refer to the derivation in Sections |3] and 0J followed by an optimization of the tilting measures 
derived in these sections. 
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Figure 7: Comparison of asymptotic growth rates of the average distance spectra of ensembles of 
RA and ARA codes. 
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Figure 8: Attainable channel regions for the rate one-third uniformly interleaved accumulate-based 
ensembles with puncturing depicted in Fig. |SJ These regions refer to the asymptotic case where 
we let N tend to infinity. The communication takes place over J = 2 parallel binary-input AWGN 
channels, and the bits are equally likely to be assigned over one of these channels (ol\ = a.2 = 
The achievable channel region refers to optimal ML decoding. The boundaries of these regions 
are calculated via the generalization of the 1961 Gallager bound followed by the optimization of 
the tilting measures (see Section 0}. The capacity limit and the attainable channel regions which 
corresponds to the cutoff rate are given as a reference. 




Figure 9: Accumulate code with puncturing period p = 3 and an equivalent version of an SPC(p) 
code followed by an accumulate code. 
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